LLM Part 1: Tokenization

 Hi everyone!

As promised, I will present a tutorial on how to build your very own large language model!  This part focuses on tokenization.

What is tokenization?

It's the process of converting human text into a sequence of numerical IDs that LLMs can understand.  A similar idea among humans is word-by-word translation of individual words from one language to another.  

For example, a word-by-word translation of "I love you." into Japanese can become "私（I）愛（love but this word is in noun-form）君（you）."

(Important: Not including context or meaning at all)

Tokenization is a necessary first step because how would a model do what you want if it doesn't even know what words you're saying!

How does tokenization work?

1. Prepare text
2. Break down text into smaller chucks of text:  These can be as following
1. Words
2. Alphabets
3. Sub-words (Common letter combinations in words) - Most common pattern
3. Convert chunks into numbers called token IDs

That's it!  Tokenization is all about creating a series of token IDs from your text.

To get a better understanding, I prepared a mini-project tokenizing and de-tokenization the text "Hi! I'm Lukas Fleur.  Nice to meet you."

LLM Part 1: Tokenization Code Link

Ending

How did you find this first part?  When I was reading about tokenization for the first time, I felt like I had to read around in circles to get a good understanding of it.  Of course, if you want to have more in-depth knowledge there are more resources out there, but this is a good first step in understanding tokenization, and thus LLMs.

For now, we'll be focusing on building our own T5-model but we can revisit the topic of tokenization in the future to see what we can do using this technique!

How did you find this article?  If you enjoyed it, or hated it, comment down below!

Resources

T5: Legacy

T5: Updated

Introduction: What are Large Language Models?

Happy New Year!

Welcome back!

At the end of 2023, I started mentioning ChatGPT.  

Merry Christimas! ... Also my thoughts about GPT

To show what makes GPT models so special, it would be interesting to give a basic introduction to large language models (LLMs) in general, focusing on building simple LLMs using T5 models.

You may be wondering "What is LLM?", "What are T5 models?", "You talked about GPT last month, so why use T5 models instead?"

What is LLM?

LLM refers to a type of AI that can generate text-based answers.  ChatGPT uses GPT models which are a type of LLM.  You have your LLM, you ask it a question, and it gives you back an answer in text.  

What are T5-models?

T5 models are another type of LLM.  

You talked about GPT last month, so why use T5 models instead?

• Intuitively easier to understand LLM concepts:  I'll write about this in more details in future blog posts, but building T5 models require a basic understanding to how LLM reads, understands, and generates human text.  T5 models are SUPER easy to build so you can have the satisfaction of building your own AI while also learning how they work at the same time!  GPT models are very easy to use but most of how the LLM generates text isn't made apparent.  (GPT models are really cool in this aspect but more on this in the future.)
• Cost: OpenAI gives an initial $5 credit to use OpenAI models, including GPT models, but after that's spent we have to pay to use them.  T5 models are completely free to build!  
• Simple coding:  Basic T5 models only require a few lines of code!  

Starting next month, I'll be introducing key concepts related to how LLMs generate text:

• Tokenization
• Encoding
• Decoding

Don't know what these mean?  Stay tuned!  On February, we're going to be learning "tokenization."  Get ready to build your own LLM!

Have any questions?  Any requests?  Do you just want to chat?  Comment down below!

Merry Christimas! ... Also my thoughts about GPT

Merry Christmas and may you all have a Happy New Year!!!

As I'm starting my end-of-year vacation, I wanted to address my readers about how I'm planning to continue my blog in the future.  As much as I enjoy writing for you all, I realized that I prefer being spontaneous with my blog posts since I live a fairly structured life outside of blogging.  Instead of racking my brain with a new topic to write about every month, and getting stressed because I couldn't stick to my schedule, I'm going to continue taking a loose approach to writing.  

If you're not already aware, I work as a data scientist building and researching machine learning models applicable for improving existing services or developing new solutions.  I'm currently working on GPT models, including the latest GPT4, so I figured I could write how I feel about GPT for today and the foreseeable future.

In terms of asking GPT to perform a certain task in general, I believe that GPT is a very useful tool to help structure your writing, provide rough translations, or produce reasonable summaries, or really any task involving the written language.  However, I do believe that objectively testing how accurate the information is and how well GPT adheres to specific additional restrictions remains a difficult issue that will be a central research topic.  I enjoy the flexibility of being able to utilise prompting methods more flexibly using the programmatic interface rather than ChatGPT or Playground.  Due to the boom of ChatGPT, as well as other AI tools, I figured that you be interested in reading about such tools from someone that is currently paid to research and refine them 👀 

Hopefully you will look forward to future posts... when they come out XD  

How do you feel about ChatGPT?  Do you find it to be a useful and innovative tool to solve your problems or improve your work?  Let me know in the comments below!

Again, Merry Christmas and have a lovely New Year! 

Autistic in Tech: The Good, the Ugly, and the Ableism

LUKAS HAS RETURNED 

Hi everyone!

I know it's been a ridiculously LONG time since my last post, but working full-time and balancing other work and life-related activities has made my life pretty hectic 😅

Last time I wrote about my experience in going through interviews for job hunting.  Now that I've been at the office for a few months now, I figured that it's time for an update.  

It is a sort of stereotype to imagine autistic people working in tech, but do I feel like tech is actually a good place to work for autistic people in real life?  

Note: These are my own experiences and opinions, and not a full featured research study.  If you have any experiences of your own that you'd like to share, comment below!

The Good

Flexible working environment

Clothes, working hours, days off... Tech can be really flexible which is important for autistic people that have diverse needs.  I could wear anything from a suit to a hoodie at work, although it seems like I can't wear jeans to work for being "too casual."  I knew someone that wore sunglasses at work because of sensory sensitivity issues, and while some may have found it odd, this person seemed happy since they could continue doing so.  When it comes to working hours, while there are core hours I have to adhere to, beyond that I can clock in and clock out whenever I want as long as I attend mandatory meetings.  If there's any sort of technology I want or need to use, I can ask someone from the company for permission and it's usually possible.

Diverse personalities and interests

My workplace seems to take great pride in welcoming many types of people.  While I do think that in part, this can be some kind of corporate rhetoric to "encourage diversity," at the very least I feel like there's little urgency for conformity when it comes to how I think or what I like.  Everyone has their own interests and contribute some thought during discussions.  Hanging out with other colleagues (e.g. drinking parties) isn't mandatory unless on special occassions but you can go out if you want to.

Office and Remote working

Somewhat related to the flexible working environment, due to the pandemic, there's a more prevalent infrastructure for remote working.  Tech-related tasks can usually be done at both the office and at home.  It's possible to go to the office everyday, and it's also possible to work from home most of the time.  It really depends on the individual's circumstances.  

Investment in physical and mental health

Tech jobs practically force employees to sit in front of a computer screen for hours and hours.  This makes the person more susceptible to poor posture, less exercise, and poor sleep.  Perhaps this is the reason why my company takes great measures to give employees a chance to maintain their health by hiring HR staff that specializes in mental health in the workplace and having healthcare plans specifically catered toward the IT industry.  I find that this is beneficial for everyone, but since autistic people are said to have a higher tendency to have health issues compared to other populations, this can be especially important to consider.

Good salary 

Jobs in tech companies tend to offer fairly decent to great salaries.  I don't personally recommend finding your dream job with the money alone, but it is something to consider.

The Ugly

Long screen time 

It really is part of the job description.  You sit in front of a computer and work on finding solutions through coding, preparing presentations, and attending meetings.  At the company where I work, almost everything I do involves the computer.  It becomes difficult to make sure to take care of your eyes and make sure to take regular breaks to avoid health issues in the future.

Male-dominated field 

Maybe in certain parts of tech, there are more women who are actively involved in tech positions.  While there are some women who have held successful careers, I've noticed an overall lack of female tech workers in my company with less than 10% of tech positions in my company held by women!  I do believe that my company tries to be female-friendly as best as they can, and the gender ratio doesn't really affect my day-to-day life.  However, there have been instances where I the women in my company have to do certain types of jobs more than our male counterparts.  It's most noticeable when the company attempts to recruit new workers, and especially if at least one of the recruits is female, and the HR department prefers having a female tech employee to emphasize how female-friendly the company is.  Although at least in my experience, the women where I work at are fairly close.

Social interactions continue to be tricky

Apparently there's a stereotype that there's a lot of autistic people in tech.  While certain autistic traits are more-or-less acceptable in the tech space, there are many employees with varying backgrounds.  With different backgrounds comes different opinions and ideologies and sometimes these can clash.  Unfortunately, "reading the room" and following social norms continues to be a requirement for maintaining a peaceful work environment.  

The Ableism

Disability shouldn't be a "dirty word,"...  but it is (?)

It's interesting to me that certain autistic traits are accepted, especially the ability to have a sharp focus and deep interest in the topic, but disability conversations are not as commonplace as what I've been used to in university where we had a disability student society as well as a disability support centre.  Sometimes I wish we could have more meaningful conversations about health and accessibility since I've felt health and safety meetings are treated as a chore.

What counts as "reasonable" accommodations?

When I was in university, there was a disability support centre where they had a website with a list of disabilities and some example accommodations.  Each student was also assigned a tutor to help navigate reasonable accommodations.  In the workplace though, it's HR that investigates anything to do with employee welfare.  The company also doesn't have an open policy about disability besides revealing that there is an insurance plan in case someone becomes disabled due to the job.  Thankfully my workplace is fairly flexible in terms of working hours and I can request adding extra functions to my office computer, but I still believe that information about disability support can be made more widely available for the public eye.

The lack of disability-related education in some people can be... astounding...

Once, there was someone that gave a talk saying that "a disabled person won't be able to do the job."  Needless to say I was dumbfounded when I heard this but it also enlightened me that some people have never thought about disability and disabled people's potential in the workplace.  

Thanks for reading!!!

This post has taken me wayyyyyyy to long to write!!!  Thanks for reading until the end.  I hope you enjoyed it, and please feel free to leave any comments about the topic.  How do you feel about being autistic in tech?  Do you have different experiences and opinions?

I honestly have no idea when I'll be writing next but till next time!  Have a great life!

Autistic person talks about job interviews

Hi everyone!

Earlier this month on Twitter, I asked my fellow Autistics people there what part of job hunting they found difficult.  The majority voted for "Interviews", so for this month, I'd like to share my experiences in interviews.  Hopefully you can find some solace or enlightenment from this post 😊

Content:

1. An overview of the job application process
2. Why do we need to have interviews anyway?
3. Types of interviews
4. When did I feel like the interview went well?
5. When did I feel like the interview went... not-so-well?
6. Did my feeling and the results match?
7. My overall thoughts on interviews
8. Dear fellow autistic people 
9. Final thoughts

An overview of the job application process

Interviewing is a common method of trying to find "the one" potential employee but it's still usually just one part of the job application process.  In most of the companies that I applied for, the application usually consisted of the following stages.

1) Preparing/Submitting application forms (CV, resume, portfolio, other documents)

2) Aptitute test (usually testing basic literacy, numeracy,  and "logical thinking")

3) Interviews (usually 2-3 rounds but it is possible to have more or less)

4) Offer (Hopefully 🤞)

There can also be take-home assignments, assessment centres, casual meetings with human resources (HR) or other employees depending on how competitive the role is and how much the company wants to assess "cultural fit." 

What does cultural fit mean?  Essentially it means that the person's presentation and core values align with the vision and values of the company.  The point of finding people who are a good cultural fit seems to stem from the idea that the company would have the manpower to realize their business plans and create their version of the ideal workplace, and keep their employees happy and keep working for the company.  (Sources:  Business News Daily, BBC) 

Due to the pandemic, most of the job application process (or at least for me) was online in the comfort of my own room, but I did have to go to the office in-person for a few occassions.

Why do we need to have interviews anyway?

From the company's perspective, I believe that interviews specifically would be considered a viable option to assess the following:

• Cultural fit
• Whether the applicant can present themselves as "professional" according to maintream corporate standards
• Whether the applicant can show that they have the skillset that the company wants (including communication skills)

It may seem like there's a power imbalance with the company having more power but I can see how the job seeker would have some use for interviews as well.  As much as the company wants to find someone that fits, so would the job seeker.  I understand that when you're seeking for a job, you can feel like any company would do but if you want to look for long-term employment, it would be desirable to be able to blend in with the working environment and build rapport with the people.  It's also a good opportunity to check whether the company... actually knows what kind of people they want to hire.... (Sometimes the job advertisement may present itself as something and the job itself being something else...)  

Types of interviews

Generally speaking, I experienced three types of interviews.

• HR 
• Usually the first interview
• Could also be a casual meeting
• Usually would look for basic back-and-forth verbal communication skills and whether you can dress and act "properly"
• Technical 
• Usually the mid-level interview
• The interviewer is an employee/manager that has a good understanding of the skills and experience required in the position
• May involve technical tests (e.g. coding tests for programmer roles)
• Would involve an in-depth assessment of the applicant's skillset
• Executive 
• Usually the final interview
• Most likely to assess cultural fit
• May ask the applicant what their career goals are

Not all companies involved all three types of interviews and in this exact order.  Some companies had more interviews and some had less, but broadly speaking these are the sort of interviews you would likely encounter.

When did I feel like the interview went well?

I would get a good feeling after an interview if I:

1) Managed to answer all the questions without getting stuck

2) If the interviewer was friendly 

3) If the interviewer wanted to present the company in a positive light when I asked questions at the end (e.g. good pay, good benefits, positive environment etc.)

When did I feel like the interview went... not-so-well?

I didn't feel so confident when:

1) I got stuck in the middle of the interview

2) If the interviewer seemed to either try to encourage (or rather console) me to "keep trying hard" or if the interviewer seemed less friendly as time went on

3) If the interviewer tried to present the company in a negative light when I asked questions at the end (e.g. too much overtime etc.)

Did my feeling and the results match?

For the most part I felt like the interview went as well as I thought it did (probably 8/10).  That being said, there were times when I was positively surprised and there were times when I thought I did well but ended up getting rejected.  There were also places that didn't even bother to send out a result (which mostly likely means a rejection but how annoying that they don't at least send out an email... lol...).  

My overall thoughts on interviews

Overall, although I felt like it took a while to land that job, I think interviews are here to stay.  It's such a mainstream method in the job application process that a lot of people have had a history of developing.  It seems to serve some purpose from both the interviewer and the interviewee, even if it feels like there is a power imbalance since one party would probably be more desperate than the other.  

My overall engagement and enjoyment of the interview largely depended on whether I could see myself getting along with the interviewer, and also whether the atmosphere of the interview made me more comfortable than nervous.  I tried not to think too much of the outcome when I was at the interview, because I thought it might make it harder to do my best.  The main parts of interviews that I disliked was that as much as I would prepare, it's almost impossible to know what kind of interviewer I would have, what kind of questions would carry more weight, or when I would ponder if I said or did the "wrong thing".  It also sucks if I can't answer all their questions regardless of whether I get the offer or not.

Each company gave me different reasons for their rejections, but it seemed like I generally struggled with showing my skillset in tech roles.  I wasn't that confident in my tech skills (programming, machine learning etc.) since I didn't have a strong computational background (No CS degree for example).  I think that being autistic meant that for me, it was even harder to make confident statements since I have a habit of spacing out mid-conversations and misinterpreting what people are saying. 

In the end though, I'd like to think that I'm going to be at a company that's right for me at present time, since I tried to be as honest as I can  (masking only the bare minimum like wearing a suit).  I got to ask my own questions for the company for over an hour after all haha!

Dear fellow autistic people

I understand that autism is a spectrum and thus there are all sorts of autistic people.  Some of you may struggle with interviews, some of you may thrive, and some may never have been interviewed yet.  Interviews can be tough for anyone, but since autistic people tend to be the minority in thought processes and communication patterns, it can be especially difficult to find that one company that will give you a chance, and one that seems like the right fit.  

From my own experience, I found that I got burnt out quite a bit since I felt like I had to mask more than usual.  I was fairly worn down towards the last couple of months.  I'm glad that I got an offer from a place where I masked the least because that gives me more confidence that I'll be okay there.  Although I suppose time will tell XD

There's no right answer I can give to "how to get a job" or "how to have the perfect interview" but my advice would be to handle job hunting as training/running a marathon.  Just keep thinking about how to present yourself, practice, do the interview, and try again.  When you've finished one interview, move on to the next one.  Take regular breaks, and maybe even longer breaks when feeling worn out.

Wherever you are, good luck!

Final thoughts

I hope you enjoyed this month's article.  There's still more that I could talk about, but I think I've written what's important.  Check out my previous posts!  My past autism related posts are:

• AUTISM AWARENESS (or ACCEPTANCE) MONTH 2021
• Is autism a disability?

What are your thoughts about job interviews?  What did you like or dislike?  Do you think being autistic has its advantages or disadvantages?  Let us know in the comments!

If you want to have a say in what I share next, look out for my Twitter!

See you next month!

Job hunting in 2022 for tech be like...

 Hi all!

I've FINALLY gotten around to writing again!!!  Missed you all loads!

A while ago, I started a poll about what topic to cover next and the majority of you chose "job hunting."

Therefore, I'm going to share my experiences finding a job in tech.  

What kind of jobs did you apply for?

In short...

• Data scientist or analyst 
• Software developer/engineer/programmer etc.
• Consultant 

After doing a research project and a temp job in academia related to big data, I realized that I really enjoy data science.  Initially when I first started job hunting, I wasn't too sure what aspect of data science I would be suited for the most.  The coding?  The statistics?  Being able to make pretty presentations?  And so on and so forth.  

That's why I applied for any tech job related to data science or big data projects.

What aspect of data science were you most equipped for job-wise and why do you think that is?

In the end, I feel like I did better when it came to applying for jobs with a heavier emphasis on "eagerness to learn", as well as having good enough presentation and communication skills to talk about myself and my academic or real-life data science projects in a logical manner.  I got a job offer for a broad tech position at a data science company and their main criteria seemed to be just that.  There was another position where I managed to get to the final interview, and that was the kind of people they claimed to have wanted.  Maybe it's because I was applying for entry-level new graduate positions, but I was honestly surprised with how important it was to show that I am eager to improve myself.

Was it important to look for specific job titles when applying?  For example, if I wanted to become a data scientist then should I only look for advertisements specifically titled "data scientist wanted?"

Actually I found that in the world of data science, it was important to have a look at the job description rather than the job title itself.  When I was applying, I specifically looked for job titles labelled data scientist because the term itself was booming online through articles and YouTube videos titled "How to become a data scientst" etc.  Eventually I realized that different companies used that title to recruit people in part because of the hype surrounding the term but in reality wanted something else like programmers for app designs.  Then I started expanding my search for any broad tech position that uses big data in some way shape or form.  It still wasn't a perfect filter for the "ideal" job but the job hunting process became more effective. 

How did you hand in your applications?

For some companies I applied through the company recruitment website or web page, and for others I applied through a recruitment agency.  This agency specifically helps students or graduates from graduate/postgraduate school find a job at a company that wants such people.  After browsing through online reviews, I felt that as someone who has a Master's degree in a STEM field looking a tech position, it would be a good fit for me.  My agent was a lovely person who helped me with every aspect of the application process (writing resumes and preparing for interviews tailored to the company) that I felt supported for the most part.  I ended up taking a job through the agency anyway.

What was the application process like?

Any combination of resume screening, interviews (online or in-person), screening tests, coding tests or take-home projects.  It really depends on the company but in my experience I usually had to hand in my resume, then take a screening test, then have 2 or 3 rounds of 30-minute to 1-hour interviews then wait for the job offer.  It might be best to share what each individual step was like otherwise this is going to be a loooooong post haha.

How many jobs did you apply for before getting a job offer?

About 30.  I managed to get to the interview stage for about 20 of them, then got to the final interview stage for 2 then got 1 job offer.

What was the toughest part of job hunting?

The rejections.  Sometimes it was because I really wanted the job, but for the most part it was because of the accumulating thoughts of "maybe I'll never find a job" and "I put so much effort... sigh..." that built up a lot of stress over time.  Towards the final couple of months, I had trouble sleeping and was gettting tired throughout the day.  Once I burst out crying in the middle of the night randomly.  If there's one thing I took from the negatives, it's TAKE CARE OF YOUR HEALTH.  The basic advice of getting regular exercise, taking care of other aspects of life as well, and taking regular breaks apparently applies to job hunting as well.  As someone who reached this state after 30 job applications, I honestly think those of you that went through hundreds of applications.... are true soldiers in the modern world!!!!

On a lighter note, did you find anything enjoyable about job hunting?

Being able to meet so many people in various businesses, companies, positions and different personalities.  It's such a wide world out there, and it was enlightening to know that there's a lot more that the universe has to offer than what I've been used to.  Even though the rejections were hard, it was also one step closer to learning about what kind of job and environment would be suitable for me and what kind of people I can get along with.  In the end, companies and industries are made up of people.  Try and find the kind of people you can see yourself working with (and hopefully enjoy working with) because it would probably make your job hunting more enjoyable.

Any final words?

The biggest piece of advice I can give, is if possible, try to find someone to job hunt with.  It can be useful for practical reasons like being able to view yourself more objectively (finding strengths or weaknesses), practicing interviewing, and for emotional reasons like being able to share your ups and downs during your job hunting experience.  Might not be nice to hear if you consider yourself to be a loner, but I honestly felt relieved sometimes knowing that I can talk to my agent or my friends and family during this time.

Is there anything else you'd like me to share?  Comment down below!

Also if you want to have a say, I'm usually most active on Twitter.

I hope you come back here for the next post!!!

Merry Christmas and a Happy New Year!!!

 Hello, my readers

It's been a while since my last post on this blog.  Honestly I wasn't sure what sort of topic I should write about and I've been pretty busy, but in the spirit of Christmas I felt to talk about a fairly big transition in my life:  Venturing into the world of finance from medicine.  If you're about to transition into a new chapter of your life, or you think you might, maybe you can find some enjoyment in this post.

If you read my About Me page, then you'll know that I completed my secondary and tertiary education in medical research (especially in neurology) with a final Master's dissertation about using big data for dementia research.  Since then I've been working in a university lab studying topics that exist on the intersection of medicine and business.  Without giving away too much due to privacy concerns, my daily tasks consist of collecting and processing data related to healthcare.  Initially I was considering starting a PhD about utilizing data science for neuroscience research, so this job was the perfect opportunity for me to develop my academic skills while I was applying for PhD places.  In the end, I did get that PhD opportunity but by that time I started feeling like I was getting complacent in academia.  I wanted to try and start a new chapter of my life into the corporate world.  Is academia really where I want to be?  Would I actually prefer working in a company if I tried it?  Is medicine what I really want to pursue a career in?  I just couldn't shake those feelings.  

After 4-5 months of job hunting, I ended up accepting a tech position at a FinTech company.  To be perfectly honest with you, when I first saw this job advertisement, I didn't think I would get the job simply because of my educational background.  One of the topics under research at the lab where I work can cover healthcare finances, but I didn't have any formal education in finance nor a very strong academic foundation of engineering.  Even my family members felt it would be difficult for me.  I still applied for the position because I was really interested in it, and I'm so glad that I did!  Apparently the company thought that I was a strong logical thinker and felt my interests and observations regarding digital monetary transactions were genuine and I would be self-motivated enough to grow myself into the role.  Due to the job requirements, I've been studying for a securities broker qualification (which I passed yay!) and I learned A LOT about money, trust funds, stocks, bonds etc.  In the past I would not have ever dreamed of being able to understand why Wall Street crashed, but now I know that it's because of messing around with the derivative market!!  That's so cool to me that I can say that now!  I still like medicine and will continue to learn and read about topics I find interesting (most likely neurodiversity), but I just love that I can still learn so much about the world and start my new journey in the fabulous world of FinTech.  

If you're at a point in your life where you feel like you want to make a change, maybe go ahead and try!  Figure out what you can do and what you need to be able to do to make those changes.  It might take a long time, it might be hard work, but it will never happen if you don't try.  Finally, rejoice the year with a Merry Christmas and may you all have a wonderful New Year!!!   

Kind regards,

Lukas Fleur

P.S.  I started advertising a magazine called North Wing Magazine (the logo in the top right hand corner).  It's a medical magazine primarily run by students hoping to spread awareness regarding medical and healthcare related topics, especially in the UK.  There's essays, articles, or simple pieces written by some amazing people!  If that's your cup of tea, check it out!