1.LLM과 sLLM 개념
■ 관련 용어
LLM(Large Language Model) 및 sLLM(small LLM)의 개념을 파악하기 위하여 관련된 태스크, 데이터, 모델 등을 아래와 같이 정리할 수 있음
-“업스트림 태스크 데이터” 용어가 너무 길기 때문에 “일반 데이터”로 명명함
– “다운스트림 태스크 데이터” 용어가 너무 길기 때문에 “도메인 데이터”로 명명함
● 모델의 가중치를 변경하는 학습 방법 및 모델은 아래와 같이 구분할 수 있음
2.LLM의 출현(emergence) 조건
LLM 모델의 출현(emergence) 또는 능력출현(emergent abilities)은 유도된 능력으로 해석될 수 있으며, 이는 개별 모델의 능력을 결합함으로써 더욱 강력한 새로운 기능이나 능력을 지닌 통합 모델이 되는 현상을 의미함. 더 많은 데이터가 통합됨으로써 새로운 능력이 암묵적으로 출현되는 것을 의미함
● LLM의 출현(emergence)이 언제 발현이 되는 지에 대한 연구가 2022년
“Emergent Abilities of Large Language Model”이란 주제로 Google
Research, Stanford University, UNC chapel Hill, DeepMind가 연구함
- 해당 논문을 참조하면(아래 그림)을 보면 대략 모델의 패러미터가 100B에 근접할수록
다양한 분야(Task)에서 성능이 향상되는 것을 알 수 있음
3. sLLM의 출현(emergent abilities)
● 최근 마이크로소프트의 연구를 보면 초거대 AI의 성능은 모델의 패러미터의 수보다는 훈련 데이터의 품질에 더 의존한다는 연구결과를 발표함.
마이크로소프트 리서치의 논문 “Textbooks Are All You Need”를 참조하면 트랜스포머 기반 1.3B 패러미터를 가진 모델을 웹으로부터 교과서 수준으로 잘 정제된 6B 토큰 데이터와 GPT-3.5로 생성된 1B 토큰 데이터를 사용하여 A100*8으로 4일 동안 학습시킨 phi-1 모델이 출현(emergent abilities)을 보여주고 있다고 설명함.
논문의 결과를 보면 1.3B에 해당하는 매개변수와 7B에 해당하는 데이터셋 크기로 학습시킨 phi-1 모델의 성능이(50.6%) 175B 매개변수 이상으로 학습시킨 GPT-3.5의 성능(47%)보다 좋은 결과를 보임.
● 이와 같이 최근에는 패러미터의 수가 무조건 큰 LLM을 그대로 사용하는 것보다는 패러미터의 수가 훨씬 적으면서 유사한 성능을 보여주는 small LLM에 대한 연구가 지속되고 있음
- 실제 LLaMa-13B는 GPT-3보다 10배 이상 작지만 더 뛰어난 성능을 보이고, LLaMa-65B는 Chinchilla-70B 및 PaLM-540B와 비슷한 성능 보여줌
- 이러한 특성으로 인해 국내외 초거대 AI 시장은 다양한 형태의 경량화 모델인 sLLM 모델 기반의 서비스 시장이 빠르게 확산되고 있는 상황임
32 Responses
I have been exploring for a little bit for any high-quality articles or weblog posts in this kind of space . Exploring in Yahoo I ultimately stumbled upon this web site. Reading this info So i am satisfied to exhibit that I have an incredibly excellent uncanny feeling I came upon just what I needed. I most no doubt will make certain to do not fail to remember this web site and give it a look on a relentless basis.
Hey there! I realize this is sort of off-topic but I had to ask. Does running a well-established blog such as yours require a lot of work? I am brand new to operating a blog but I do write in my diary on a daily basis. I’d like to start a blog so I can easily share my own experience and feelings online. Please let me know if you have any kind of ideas or tips for new aspiring bloggers. Thankyou!
I think this is among the most significant information for me. And i am glad reading your article. But wanna remark on few general things, The site style is great, the articles is really great : D. Good job, cheers
If some one wishes expert view regarding blogging and site-building after that i suggest him/her to pay a visit this blog, Keep up the fastidious work.
My coder is trying to persuade me to move to .net from PHP. I have always disliked the idea because of the expenses. But he’s tryiong none the less. I’ve been using Movable-type on a number of websites for about a year and am concerned about switching to another platform. I have heard great things about blogengine.net. Is there a way I can import all my wordpress posts into it? Any kind of help would be really appreciated!
I am really impressed with your writing skills as well as with the layout on your blog. Is this a paid theme or did you customize it yourself? Anyway keep up the excellent quality writing, it’s rare to see a nice blog like this one nowadays.
Excellent post. I was checking constantly this weblog and I am impressed! Very useful info specially the last section 🙂 I take care of such info a lot. I was seeking this particular information for a very long time. Thank you and best of luck.
Genuinely no matter if someone doesn’t know after that its up to other visitors that they will help, so here it happens.
Thanks for any other great post. The place else may just anyone get that kind of information in such an ideal means of writing? I have a presentation subsequent week, and I’m at the look for such info.
Howdy! Would you mind if I share your blog with my facebook group? There’s a lot of people that I think would really appreciate your content. Please let me know. Thank you
Hi my family member! I wish to say that this article is amazing, nice written and come with almost all vital infos. I’d like to look extra posts like this .
You could definitely see your expertise in the work you write. The arena hopes for even more passionate writers such as you who are not afraid to say how they believe. At all times go after your heart.
Hi excellent blog! Does running a blog such as this require a large amount of work? I’ve no understanding of coding however I had been hoping to start my own blog soon. Anyhow, if you have any recommendations or tips for new blog owners please share. I know this is off subject nevertheless I simply wanted to ask. Many thanks!
Definitely believe that that you stated. Your favorite justification appeared to be on the internet the simplest thing to take note of. I say to you, I certainly get annoyed while people think about issues that they plainly don’t know about. You controlled to hit the nail upon the highest as neatly as outlined out the entire thing without having side effect , people could take a signal. Will likely be again to get more. Thanks
I am sure this piece of writing has touched all the internet viewers, its really really pleasant piece of writing on building up new blog.
Ahaa, its fastidious discussion on the topic of this piece of writing at this place at this web site, I have read all that, so at this time me also commenting here.
I visited various websites but the audio feature for audio songs current at this web site is genuinely marvelous.
Excellent post. I was checking continuously this blog and I am impressed! Very helpful info specifically the last part 🙂 I care for such information much. I was seeking this certain information for a long time. Thank you and best of luck.
Have you ever considered about including a little bit more than just your articles? I mean, what you say is fundamental and all. Nevertheless think about if you added some great graphics or video clips to give your posts more, “pop”! Your content is excellent but with images and videos, this site could undeniably be one of the best in its field. Fantastic blog!
You ought to be a part of a contest for one of the greatest sites on the web. I will highly recommend this blog!
I’m really enjoying the design and layout of your website. It’s a very easy on the eyes which makes it much more pleasant for me to come here and visit more often. Did you hire out a developer to create your theme? Exceptional work!
Nice post. I was checking continuously this blog and I’m impressed! Very useful info particularly the last part 🙂 I care for such info much. I was looking for this particular info for a very long time. Thank you and good luck.
Magnificent beat ! I wish to apprentice while you amend your web site, how can i subscribe for a blog web site? The account helped me a acceptable deal. I had been a little bit acquainted of this your broadcast provided bright clear concept
It’s genuinely very difficult in this busy life to listen news on TV, thus I only use world wide web for that reason, and take the most recent information.
Hey just wanted to give you a quick heads up. The text in your article seem to be running off the screen in Ie. I’m not sure if this is a formatting issue or something to do with browser compatibility but I figured I’d post to let you know. The design look great though! Hope you get the issue fixed soon. Many thanks
Do you mind if I quote a couple of your posts as long as I provide credit and sources back to your weblog? My blog site is in the exact same area of interest as yours and my visitors would definitely benefit from a lot of the information you provide here. Please let me know if this okay with you. Cheers!
Very good blog! Do you have any helpful hints for aspiring writers? I’m planning to start my own blog soon but I’m a little lost on everything. Would you advise starting with a free platform like WordPress or go for a paid option? There are so many choices out there that I’m completely confused .. Any suggestions? Appreciate it!
Hi, I do think your web site may be having browser compatibility problems. Whenever I take a look at your web site in Safari, it looks fine however, when opening in IE, it’s got some overlapping issues. I simply wanted to give you a quick heads up! Aside from that, great blog!
It’s awesome to go to see this web page and reading the views of all friends about this paragraph, while I am also zealous of getting familiarity.
Thanks for ones marvelous posting! I certainly enjoyed reading it, you happen to be a great author. I will make sure to bookmark your blog and definitely will come back from now on. I want to encourage you continue your great job, have a nice holiday weekend!
This is really attention-grabbing, You’re an excessively professional blogger. I’ve joined your feed and look ahead to searching for extra of your fantastic post. Also, I’ve shared your web site in my social networks
I’m not that much of a online reader to be honest but your sites really nice, keep it up! I’ll go ahead and bookmark your site to come back in the future. Cheers