Whatchu Workin' On?

[PointOfNoReturnPoint]

I am working on turning this pizza into a shit post. Does that count?

 

[dvnc]

A Spring 2024 fashion Campaign. Web design and other behind the scenes stuffs. It's a fun contract.

 

[Flabba Wabba Jabba]

Trying to get into AI LLMs, I don't know if being your 30s turns your brain to mush but I'm having a really hard time understanding this alien tech crap.

Anyone else? what are you using?

I'm pretty across things, so feel free to ask anything about anything. I've worked a lot with LLMs since like 2019 for some early university research projects that were being offered to students for testing.

 

[cybertoaster]

I'm pretty across things, so feel free to ask anything about anything. I've worked a lot with LLMs since like 2019 for some early university research projects that were being offered to students for testing.

Are you working in AI right now or just doing stuff as a hobby?

 

[Blamo2000]

I want to make some game projects and a personal website. Those are in plan for 2024.

 

[Flabba Wabba Jabba]

Are you working in AI right now or just doing stuff as a hobby?

I run a company that does project work/integration which includes a LOT of LLM-related stuff (setting up local models for companies, setting up codebases to interact with OpenAI APIs, Claude 2 APIs), helping corporate developers understand LLMs and their use-case for internal projects, yada yada.

So no I don't work for an AI company, but I'm like 90% of the way there on the consumer end of things.

I want to make some game projects and a personal website

I love running my own site. I have a little forum setup where I can post (and others can too if they mysteriously decide to create an account). I really enjoy tinkering with it in my spare time.

 

[cybertoaster]

I run a company that does project work/integration which includes a LOT of LLM-related stuff (setting up local models for companies, setting up codebases to interact with OpenAI APIs, Claude 2 APIs), helping corporate developers understand LLMs and their use-case for internal projects, yada yada.

So is this a consulting/advisory type thing or your company is developing actual custom AI products based around those APIs? like are you selling them the finish product or just telling them what they can do with that?

If you don't mind me asking what kind of companies are hiring you for this? I know healthcare is going big on AI, Humana just got one to autoreject treatments, has an 80% bias to rejecting but I guess that's actually a feature and not a bug for them given the billions they're saving in the process.

Anyway, what else? law firms? accounting? trading I assume they have their own in-house department given how dependent they are on bots.

 

[Flabba Wabba Jabba]

So is this a consulting/advisory type thing or your company is developing actual custom AI products based around those APIs? like are you selling them the finish product or just telling them what they can do with that?

So our company does both, we implement but also do some advisory stuff (90% of it has been implementation). We do not develop custom AI models, we implement both API-accessible LLMs like ChatGPT, GPT-4 and Claude-2. We also do self-hosted LLM setups using VMs (much easier than Docker instances IMO) on Azure. All the other little things like setting up web server proxies, IP whitelisting, domain validation and then programmatic setups to interact with them (like micro-libraries that communicate with them (mainly in JavaScript and .NET/C#) come along with the work we do also.

If you don't mind me asking what kind of companies are hiring you for this?

It's a wild variety, so we've done 1 medical research internal LLM, 1 construction certification job for their clients, 1 tertiary education for student placements and a tourism/rental company chatbot for bookings/information. Most of them are just GPT-4 API hookups using JavaScript/TypeScript, but the medical one was a bit of a doozy since they had stringent requirements for things like SOC-2 compliance, Hippa compliance and other government regulations (they were using the LLM to interact with information about ongoing research logs with volatile masses/nuclear medicine).

A bit of a tangent but I've found that marketing does not work in this business, only networking and personal connections are key. We've beaten larger consultancies/firms in bidding for these jobs because we bid not only so much lower than them (some firm quoted like 500k for that job whereas we're doing it for like 125k), but we give a shit about the outcome. We're not going to just build to the spec and then leave it, we actually reach out and enquire about usage and if it's fit their expectations. Kindness goes a long way in this business.

Anyway, what else? law firms? accounting? trading I assume they have their own in-house department given how dependent they are on bots.

A lot of these companies have incompetent dev teams, or just a skeleton crew that manages infrastructure. A lot of these firms either use cloud-services (mainly Azure) for identity management (Entra ID) or have their own on-prem infrastructure for government regulations (medical mainly).

The kind of companies we target are just big enough to require bespoke solutions for problems they face, but not big enough to have dedicated developers in salaried roles. You'll see this a lot with non-profits, where they'll have their core team working on their given field, but have a ton of technological requirements with the inability to maintain or develop them. You'd be shocked at how many are still using SharePoint/Power Apps to run critical internal processes. We had one company using an accounting program from 2008 that couldn't run on Windows 10, so they had a Windows 7 virtual machine on their computer that was used to maintain over 300k/month in expenses and they refuse to switch over because their accounting logs are incompatible with Xero/Zoho/QuickBooks.

Hopefully that answers your questions

 

[cybertoaster]

We do not develop custom AI models

I mean that would be really expensive, easier to train an existing model, or you mean you don't train them either?

We also do self-hosted LLM setups using VMs (much easier than Docker instances IMO) on Azure.

Ever had a costumer who asked you for a hardware solution to run on-site?

like ChatGPT, GPT-4 and Claude-2

Have you tried Mistral? how did it go?

and a tourism/rental company chatbot for bookings/information

I take that's one of those 'just the API' cases? because there are already tons of companies offering those customer support chatbots, tho most are abysmal.

but the medical one was a bit of a doozy since they had stringent requirements for things like SOC-2 compliance, Hippa compliance and other government regulations

No surprise given all the high profile data leaks/theft going on with medical records.

A bit of a tangent but I've found that marketing does not work in this business, only networking and personal connections are key. We've beaten larger consultancies/firms in bidding for these jobs because we bid not only so much lower than them (some firm quoted like 500k for that job whereas we're doing it for like 125k), but we give a shit about the outcome. We're not going to just build to the spec and then leave it, we actually reach out and enquire about usage and if it's fit their expectations. Kindness goes a long way in this business.

Wait you say connections and not marketing matters but you did that job for 4 times less and added (what I assume is) long-term support too, that seems a really good deal for the client more than a matter of connections and networking, hard to pass on that offer when the other one is half a mil and "good luck with whatever happens".

Am I missing something here?

or have their own on-prem infrastructure for government regulations (medical mainly).

Did they ask you to build them an "LLM in a box" (eg: a blade with a few A100s) or just make it run in the existing hardware they got?.

You'd be shocked at how many are still using SharePoint/Power Apps to run critical internal processes.

A guy I know worked at a major chemical company still uses an old ass program from the early 1960's written in FORTRAN to control a bunch of industrial mixers. I remember the story because he told me that like 10 years ago they had to go looking among retired college professors to find someone who could look at the code to make some changes because all the guys who coded the original program were dead like a dodo.

So not that shocked TBH, the entire credit card system still runs on COBOL too, and IBM still sells brand new mainframes for that.

Hopefully that answers your questions

You went above and beyond bro, thanks!

 

[Flabba Wabba Jabba]

I mean that would be really expensive, easier to train an existing model, or you mean you don't train them either?

We've never been asked to train them and personally I don't see the point. RAG is much better than fine-tuning in my opinion.

Ever had a costumer who asked you for a hardware solution to run on-site?

Not yet, we've had one customer who told us they did, only to discover just a bunch of old Titan GPUs that were used for data analysis and modelling, so that was not a fun call telling them that it can't go ahead and the project was dead in the water.

Have you tried Mistral? how did it go?

It's good, I would say it's comparable to Claude 2 when properly prompted.

I take that's one of those 'just the API' cases? because there are already tons of companies offering those customer support chatbots, tho most are abysmal.

Yeah these guys just wanted a chatbot so that customers could ask simple questions instead of forking out a ton of time/money for some bespoke booking management. They were also very cheap and didn't want to pay for something like Intercom so we built just a basic chatbot with RAG. Nice little web job.

No surprise given all the high profile data leaks/theft going on with medical records.

Yes that's what their main concern was. They're the main research group for a lot of hospitals and turn over a TON of research grants, so it would be disastrous if all that got leaked.

Wait you say connections and not marketing matters but you did that job for 4 times less and added (what I assume is) long-term support too, that seems a really good deal for the client more than a matter of connections and networking, hard to pass on that offer when the other one is half a mil and "good luck with whatever happens".

Am I missing something here?

Ok I worded this terribly. A lot of the time cost is not actually the main factor in big client decision making (in my experience), it's about if you can trust them to do the job properly. We do not compete on price as it makes you appear comparable to other vendors, so a potential client will then just pick whoever has a deeper track record. We have been around for a couple of years and do not have hundreds of clients/use-cases behind us. We have about 10.

In order for us to compete, we offer things that big firms cannot, such as the intimate relationship that you get with a smaller group, especially when it comes to out-of-hours contact, holiday contact (we work every day of the year and have had failures occur on weekends/important holidays), on-call work (we also do not bill for calls involving problems with the project), training their teams about its operation and proper project documentation (unlike most of our competitors who only show what the new system is and how it works, not the underlying issues that may occur + troubleshooting and when to contact us).

The price was really just because we are a small team and will not charge the same price that a big consultancy would charge, because they would throw 4x the people at it anyway. We've had about 2-3 jobs go our way because I literally just said, "These guys will have too many cooks in the kitchen" sparking a laugh/chuckle amongst the procurement agents who then recommend us because we were nice people to talk to. These guys will risk a smaller firm if they're nice to talk to, it's amazing how far you can get by not being a soulless wagie/actually caring about the problem your client faces.

Did they ask you to build them an "LLM in a box" (eg: a blade with a few A100s) or just make it run in the existing hardware they got?.

Mentioned above and obviously that did not work, but yeah we're doing a job right now which involves us setting up a self hosted instance of Mistral on their brand-spankingly new DELL EMC E9680 server, it's a fucking monster and I'm so excited to SSH into it and just admire the specs, it's cost them like a million bucks or something absurd to purchase. It's designed to power their internal AI applications that they're building for their different teams (from what I've heard it's mainly internal customer support for employees and phasing them out LOL).

You went above and beyond bro, thanks!

Hey no worries man, always happy to share what I can. I really enjoy the unique problems that we face and the solutions we can implement. Hopefully we can get to a point where I can simply exit and make some solid money, but it's fucking brutal when you're small with 5 guys and all under the age of 30.

Oh and a word of advice. If you ever decide to get into software consulting/consulting or any fucking business in general, the cheap clients are the WORST. Cheap clients will always attempt to stretch their dollar as far as the eye can see, so if it's a fixed price job they will always try and sneak things in. If they do this, always respond with, "Ok we will revise the project plan and include this new feature in our technical specification, which will change the finished cost." This will shut that idea down immediately. Expensive customers are great because they usually have their shit together (there are exceptions!). So ideally recognise a cheap customer early on and keep a tight grip on the project, otherwise you will be doing unpaid work and it sucks. Unfortunately you will have to do this sometimes when starting out as the only people willing to bet on you are cheap customers.

Also, don't be afraid to tell a customer it's not going to work. We've turned down maybe 3 possibly enormous jobs because we were out of our league entirely or the client was delusional about what could be achieved with their budget. A lot of the time they will become frustrated or claim you to be incompetent but you can reassure them by stating, "we can always build it for you, but there may be unforeseen problems that blow the budget out." This also shuts things down since you're reminding them who's paying LOL (you can always tell a scumbag consultant if they say yes to everything).

 

[Prj4MW8iPT3vmFM2rzDc]

I just got local LLM AI working on my personal desktop using stuff from Mistral AI/ ollama with dolphin-mixtral model.
To break this down into easier to understand non-techy terms.


Imagine stable diffusion with the same ability to swap out custom trained models but instead of art generation, its like chat gbt 3.5 to 4.0 level functionality for text prompting.

 

[SleepingBull]

One of the ways I've been killing spare time at work lately has been toying around in Godot. For all of my time earning software degree and my time formerly working in the field, I never actually did any fun visual stuff. So the basic path of the project started as just a hello world of boxes in a circle, which then became a tube, and then I decided I wanted the tube to curve. It is no longer composed of boxes, now its calculating out all the primitive triangles and creating the mesh as demanded, when handed a 3d bezier curve, a number of sides, rings, and a girthcurve for input.

I have created one hell of a mesmerizing screensaver, if nothing else.

 

[Car Won't Crank]

I am working on a project. That is all I can say.

 

[Creative Username]

- snip -

Kind of looks like one of those xscreensaver programs.

... Add some cubes or something.

 

[IamnottheNSA]

I was working on getting Windows 98 to run on a modern machine. No VM, on bare metal.
There it is.

Installing and running it on anything newer than a Socket 775 machine is pretty tricky and requires a fairly specific hardware and software combination. It took me awhile to get it right, but eventually I was successful.

 

[WelperHelper99]

I recently got this old first gen blu ray player at a thrift store for 10 bucks. It has component, composite, S video, and HDMI. I have a CRT that can play at full 480p. 20 bucks later getting a remote and power cable, hooked it into the S video port, and damn, it looks really, really good. Subtitles look great, I'll tell you that much.

 

[Lone Wandering Courier]

I was working on getting Windows 98 to run on a modern machine. No VM, on bare metal.
There it is.
View attachment 5602042
Installing and running it on anything newer than a Socket 775 machine is pretty tricky and requires a fairly specific hardware and software combination. It took me awhile to get it right, but eventually I was successful.

Honestly, that's pretty badass. At one point I considered doing this, but I decided that it wasn't really worth it and that it was cheaper to just build a retro pc for that sort of thing (even though early 2000s hardware has gone up in price but I did secure a decent deal a long time ago).

 

[seri0us]

I recently got this old first gen blu ray player at a thrift store for 10 bucks. It has component, composite, S video, and HDMI. I have a CRT that can play at full 480p. 20 bucks later getting a remote and power cable, hooked it into the S video port, and damn, it looks really, really good. Subtitles look great, I'll tell you that much.

That's nice. I never moved passed DVD and while today I'm not exactly pleased with how it looks blown up on my "Full HD" monitor, though I do appreciate having a physical copy.

 

[WelperHelper99]

That's nice. I never moved passed DVD and while today I'm not exactly pleased with how it looks blown up on my "Full HD" monitor, though I do appreciate having a physical copy.

Honestly I love Blu-ray and physical media in all forms. Have tapes still, and on the opposite end 4k discs, for a better screen of course. Streaming and digital is alright, it's just the bit rate issue that blows

 

[Byte Doctor]

I Finished Making my Led Cube now, So now it's time to start experimenting with My FPGA Boards again. So after New years seems like I'll be up and at it again once the holidays are over.