Weird and Cringe things you've seen while working in IT - Since everyone is too lazy to make such a thread where IT bros can vent

[Art of Quar]

Oh fuck yeah of course. You still put in authentication measures no matter how safe it is to expose it. I guess you'd have an interoperability layer between your back-end and your database? No combination software that does both tasks?

One was an RMM and the other was a CRM/ticketing platform.

In particular, it was an RMM that just this month got exploited and caused a tech crisis.

 

[KokoroKoroki]

Not my story, but one of my friends was checking the torrent downloads of students on our university campus using the "I Know What You Downloaded" website and found out that someone had been using the free uni WiFi to download a bunch of incest hentai on their computer.

Never found out if they were reprimanded for it, but I doubt the staff cared enough to actually do something.

 

[mister meaner]

Big companies with "fuck you" money. Someone calls me in, $120 an hour, to fix their problem. What is the task, Outlook/Exchange shit the bed again because someone keeps a 30GB inbox or do they need someone that signed the NDAs to take instructions from qualified people to repair the printer? Turns out the mouse ran out of batteries, diagnosed in five seconds, client will be charged for the whole hour while they take a 45 minute break so their computer can be fixed. It was a good source of income but it was fucking stupid and things like that happened so often. Women in the workplace was a mistake.

Reminds me of the time when i was called on site to unplug a computer and plug it back in. They spent a lot of money for that.

 

[Kiwi Lime Pie]

Not my story, but one of my friends was checking the torrent downloads of students on our university campus using the "I Know What You Downloaded" website and found out that someone had been using the free uni WiFi to download a bunch of incest hentai on their computer.

At my aforementioned first IT job, internet when I first got there was limited to a company AOL dial-up account for essential use only (this was the late 90s for context). Eventually, the company opted to get a business DSL line. Internet access was initially limited to a few as part of the trial run before everyone had it. My coworker who was the programmer with the most seniority was part of the pilot group. Several days later, one of the company execs went to his desk and said something along the lines of, "I see you've been browsing the North Korean government web site."

Considering my coworker's mindset and political leanings, this didn't surprise me one bit. It also made it clear the company was already monitoring the fledgling network.

 

[Lathe]

I am a PC hobbyist who fixes my own problems. I had a friend’s company computer stop recognizing their speakers as an audio output. I fixed it the first time by rolling back the driver. The second time it happened I couldn’t roll back the driver. I told them to tell their IT dept to try deleting and reinstalling the driver first. After two days of back and forth with their IT dept, they did what I suggested. To be fair, the tech was a woman.

 

[Kiwi Lime Pie]

One of my most concerning IT incidents, ironically happened since i've temporarily shifted over to security instead of IT.

So yeah, keep in mind that guy is essentially in charge of your physical safety and the confidentiality of your patient info when you're at the hospital, and is clearly shit at both

If this in the US, this sounds like it would be a huge issue with HIPAA -- especially if any private information were to be improperly disclosed via lookup at a poorly-secured terminal.

The latter part of your story, @WonderWino, shows that this supervisor is like many who are quick to blow up over something minor (letting in a pregnant woman yourself when the other guard is away from the station in your story) while showing no sense of urgency for something far more serious (the data security issues).

tldr: he gave me a dressing down for literally doing my job.

That always sucks. I believe have a story posted in one of the horror threads about a previous boss who gave me a written reprimand for not doing my job right even though I was following nearly to the letter the written job description she herself had given me about a week earlier. I believe I turned in my two week notice the next day, and I had no regrets whatsoever about leaving.

I told my friend about this, that his dad must have been searching some wild shit. He got so defensive and incredulous that he actually became angry. I was just like whatever dude, then did what I was being paid for.

That sounds like the tail-end of my earlier post in which a schoolmate took enough issue with my good-natured ribbing after I found porn cookies in her browsing cache and stopped actively contacting me for anything after that day. People almost have to expect some sort of teasing when porn is found on one of their computers even if it's a nothingburger situation worth nothing more than a good laugh.

There are actually a surprising amount of websites that are just making some web request through js to an api that just takes straight SQL queries (I mean literally just an entire select statement).

I think this is part of the reason we're seeing so many data breaches with online stores and so many high-profile ransomware attacks.

With "time is money" being important above everything in today's business environment, people in charge are rushing to get their online/networked systems up and running so quickly that security is an afterthought or not thought about at all until it's too late. It also seems like businesses are unaware of the principle of least privilege when it comes to systems in that an average user shouldn't be given access to information/systems they have no need to be accessing. For example, an office assistant likely has no need to access production systems and an accountant probably has no need to be poking through the employer's secretarial data. Unfortunately, many workplaces don't get this concept and their networks will allow nearly unfettered access to anything with minimal at best security that won't stop someone determined enough to go exploring or cause havoc.
(E: spelling)

 

[Puff]

Reminds me of the time when i was called on site to unplug a computer and plug it back in. They spent a lot of money for that.

People pay a lot to shuffle around liability. Make my living doing it.

 

[tehpope]

I work remote IT. I see offices still have 32 bit versions of Windows running. Mostly 7 and 10. I've once seen someone still using XP.

 

[Art of Quar]

I work remote IT. I see offices still have 32 bit versions of Windows running. Mostly 7 and 10. I've once seen someone still using XP.

You dont deal with enough manufacturing plants. SCADA is expensive to upgrade so i once had to support a two domain network so their C&C machines had an NT4 domain controller to talk to.

The NT4 and 2000 servers were baremetal with no backups

 

[Celebrate Nite]

I've once seen someone still using XP.

PC Richards' corporate building here on long island still has an XP computer they use specifically for scanning documents into their system. That company is completely outdated though... their POS system is STILL from the 1980's with complicated menu trees that requite an additional "F11 - F20" to navigate to (no cheat sheets either.. have fun getting lost). Not to mention their servers use those tiny rectangle tapes to save their data that needs to be rotated like every day or every other day.

Only thing "new" they do is have all their users use a custom windows on a VMWare that you can't do ANYTHING on except word/excel and their proprietary software.

 

[Smaug's Smokey Hole]

PC Richards' corporate building here on long island still has an XP computer they use specifically for scanning documents into their system. That company is completely outdated though... their POS system is STILL from the 1980's with complicated menu trees that requite an additional "F11 - F20" to navigate to (no cheat sheets either.. have fun getting lost). Not to mention their servers use those tiny rectangle tapes to save their data that needs to be rotated like every day or every other day.

Only thing "new" they do is have all their users use a custom windows on a VMWare that you can't do ANYTHING on except word/excel and their proprietary software.

Speaking of old stuff, a friend of mine was at a place installing a completely new computer system and servers. The end users were people that had been programming satellite software for a long time. The new system environment were built around NT4 in 1996 I think.

When everything was done he showed them how to login and set a password, then he would be showing where the programs were and all that. But being a jovial nerd he jokingly asked "should I show you how to use the mouse as well?" while jiggling it around with a smile during the login process.

"Yes please, that would be very helpful."

These men and women were far from retarded it's just that they had been programming on old terminal-style computers for a long time, they had little to no experience with a mouse and they weren't teenagers that would figure it out so they could get into chatrooms. It's easy to miss something like that by assuming that smart people know simple plebian things.

When he got home his psycho borderline girlfriend would beat him relentlessly but would only leave bruises that clothes covered up.

 

[Spooky Bones]

This thread reminds me of this collection of stories from, wow, almost 30 years ago now lol. It's an entertaining read and an insight into what things were like for us old people back in the dinosaur era.

My favorite story from the collection and this guy is a fucking hero

Have you ever left your terminal logged in, only to find when you came
back to it that a (supposed) friend had typed "rm -rf ~/*" and was
hovering over the keyboard with threats along the lines of "lend me a
fiver 'til Thursday, or I hit return"?  Undoubtedly the person in
question would not have had the nerve to inflict such a trauma upon
you, and was doing it in jest.  So you've probably never experienced the
worst of such disasters....

It was a quiet Wednesday afternoon.  Wednesday, 1st October, 15:15
BST, to be precise, when Peter, an office-mate of mine, leaned away
from his terminal and said to me, "Mario, I'm having a little trouble
sending mail."  Knowing that msg was capable of confusing even the
most capable of people, I sauntered over to his terminal to see what
was wrong.  A strange error message of the form (I forget the exact
details) "cannot access /foo/bar for userid 147" had been issued by
msg.  My first thought was "Who's userid 147?; the sender of the
message, the destination, or what?"  So I leant over to another
terminal, already logged in, and typed
        grep 147 /etc/passwd
only to receive the response
        /etc/passwd: No such file or directory.

Instantly, I guessed that something was amiss.  This was confirmed
when in response to
        ls /etc
I got
        ls: not found.

I suggested to Peter that it would be a good idea not to try anything
for a while, and went off to find our system manager.

When I arrived at his office, his door was ajar, and within ten
seconds I realised what the problem was.  James, our manager, was
sat down, head in hands, hands between knees, as one whose world has
just come to an end.  Our newly-appointed system programmer, Neil, was
beside him, gazing listlessly at the screen of his terminal.  And at
the top of the screen I spied the following lines:
        # cd
        # rm -rf *

Oh, shit, I thought.  That would just about explain it.

I can't remember what happened in the succeeding minutes; my memory is
just a blur.  I do remember trying ls (again), ps, who and maybe a few
other commands beside, all to no avail.  The next thing I remember was
being at my terminal again (a multi-window graphics terminal), and
typing
        cd /
        echo *
I owe a debt of thanks to David Korn for making echo a built-in of his
shell; needless to say, /bin, together with /bin/echo, had been
deleted.  What transpired in the next few minutes was that /dev, /etc
and /lib had also gone in their entirety; fortunately Neil had
interrupted rm while it was somewhere down below /news, and /tmp, /usr
and /users were all untouched.

Meanwhile James had made for our tape cupboard and had retrieved what
claimed to be a dump tape of the root filesystem, taken four weeks
earlier.  The pressing question was, "How do we recover the contents
of the tape?".  Not only had we lost /etc/restore, but all of the
device entries for the tape deck had vanished.  And where does mknod
live?  You guessed it, /etc.  How about recovery across Ethernet of
any of this from another VAX?  Well, /bin/tar had gone, and
thoughtfully the Berkeley people had put rcp in /bin in the 4.3
distribution.  What's more, none of the Ether stuff wanted to know
without /etc/hosts at least.  We found a version of cpio in
/usr/local, but that was unlikely to do us any good without a tape
deck.

Alternatively, we could get the boot tape out and rebuild the root
filesystem, but neither James nor Neil had done that before, and we
weren't sure that the first thing to happen would be that the whole
disk would be re-formatted, losing all our user files.  (We take dumps
of the user files every Thursday; by Murphy's Law this had to happen
on a Wednesday).  Another solution might be to borrow a disk from
another VAX, boot off that, and tidy up later, but that would have
entailed calling the DEC engineer out, at the very least.  We had a
number of users in the final throes of writing up PhD theses and the
loss of a maybe a weeks' work (not to mention the machine down time)
was unthinkable.

So, what to do?  The next idea was to write a program to make a device
descriptor for the tape deck, but we all know where cc, as and ld
live.  Or maybe make skeletal entries for /etc/passwd, /etc/hosts and
so on, so that /usr/bin/ftp would work.  By sheer luck, I had a
gnuemacs still running in one of my windows, which we could use to
create passwd, etc., but the first step was to create a directory to
put them in.  Of course /bin/mkdir had gone, and so had /bin/mv, so we
couldn't rename /tmp to /etc.  However, this looked like a reasonable
line of attack.

By now we had been joined by Alasdair, our resident UNIX guru, and as
luck would have it, someone who knows VAX assembler.  So our plan
became this: write a program in assembler which would either rename
/tmp to /etc, or make /etc, assemble it on another VAX, uuencode it,
type in the uuencoded file using my gnu, uudecode it (some bright
spark had thought to put uudecode in /usr/bin), run it, and hey
presto, it would all be plain sailing from there.  By yet another
miracle of good fortune, the terminal from which the damage had been
done was still su'd to root (su is in /bin, remember?), so at least we
stood a chance of all this working.

Off we set on our merry way, and within only an hour we had managed to
concoct the dozen or so lines of assembler to create /etc.  The
stripped binary was only 76 bytes long, so we converted it to hex
(slightly more readable than the output of uuencode), and typed it in
using my editor.  If any of you ever have the same problem, here's the
hex for future reference:
        070100002c000000000000000000000000000000000000000000000000000000
        0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f
        8800040000bc012f65746300

I had a handy program around (doesn't everybody?) for converting ASCII
hex to binary, and the output of /usr/bin/sum tallied with our
original binary.  But hang on---how do you set execute permission
without /bin/chmod?  A few seconds thought (which as usual, lasted a
couple of minutes) suggested that we write the binary on top of an
already existing binary, owned by me...problem solved.

So along we trotted to the terminal with the root login, carefully
remembered to set the umask to 0 (so that I could create files in it
using my gnu), and ran the binary.  So now we had a /etc, writable by
all.  From there it was but a few easy steps to creating passwd,
hosts, services, protocols, (etc), and then ftp was willing to play
ball.  Then we recovered the contents of /bin across the ether (it's
amazing how much you come to miss ls after just a few, short hours),
and selected files from /etc.  The key file was /etc/rrestore, with
which we recovered /dev from the dump tape, and the rest is history.

Now, you're asking yourself (as I am), what's the moral of this story?
Well, for one thing, you must always remember the immortal words,
DON'T PANIC.  Our initial reaction was to reboot the machine and try
everything as single user, but it's unlikely it would have come up
without /etc/init and /bin/sh.  Rational thought saved us from this
one.

The next thing to remember is that UNIX tools really can be put to
unusual purposes.  Even without my gnuemacs, we could have survived by
using, say, /usr/bin/grep as a substitute for /bin/cat.

And the final thing is, it's amazing how much of the system you can
delete without it falling apart completely.  Apart from the fact that
nobody could login (/bin/login?), and most of the useful commands
had gone, everything else seemed normal.  Of course, some things can't
stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but
by and large it all hangs together.

I shall leave you with this question: if you were placed in the same
situation, and had the presence of mind that always comes with
hindsight, could you have got out of it in a simpler or easier way?
Answers on a postage stamp to:

Mario Wolczko

 

[The Token Ethnic]

I briefly worked in IT and had a shitty job working at an inner-city college prep school. They always had hand-me-down technology that constantly broke down or needed some sort of drivers installed for their interactive blackboards. Outside of a few teachers and the other IT dudes, we were the only non-black people there. Anyhow, a lot of these teens would send their laptops to us since they hadn't set up any web blocking programs and they downloaded tons of shit. One kid dropped off his laptop since malware took over his old Gateway laptop. Once I started looking, I found out he had tons of really nasty twerking videos, bukkake clips and the equivalent of rap AMVs set to horror movie gore clips. Me and the boys had a laugh and gave the kid who came to pick up his some flack for his choices and we got in trouble for it. Fuck that school and that kid for not having a sense of humor.

 

[Vyse Inglebard]

Two of my past jobs didn't even use ticketing systems, they just had a separate outlook inbox specifically for IT and everyone in the IT department had to watch it and mark what they were working on. Half the time they can't even be bothered to do that and just physically look for you. I got yelled at a few time for helping people that way because in one of the jobs they had a very strict "no walk-ins allowed" policy.

Those two companies should be nuked from orbit. I cannot even imagine doing my current job without a ticketing system, and as for that "no walk-ins" policy, what if there was a problem with the hardware? Are the customers just.....fucked?

Not my story, but one of my friends was checking the torrent downloads of students on our university campus using the "I Know What You Downloaded" website and found out that someone had been using the free uni WiFi to download a bunch of incest hentai on their computer.

Never found out if they were reprimanded for it, but I doubt the staff cared enough to actually do something.

A lot of porn related stories in this thread. I guess that Avenue Q song really is true:

People almost have to expect some sort of teasing when porn is found on one of their computers even if it's a nothingburger situation worth nothing more than a good laugh.

Seriously. There's no point in getting indignant about it. Practically everybody consumes porn nowadays (arguably for the worst).

When he got home his psycho borderline girlfriend would beat him relentlessly but would only leave bruises that clothes covered up.

Poor dude. Hope he's doing OK nowadays, man, holy shit.

 

[Celebrate Nite]

Those two companies should be nuked from orbit. I cannot even imagine doing my current job without a ticketing system, and as for that "no walk-ins" policy, what if there was a problem with the hardware? Are the customers just.....fucked?

Let's put it this way... with the company that had the "no walk-ins allowed" policy, not only was I working in IT, but I was also THE shipping guy for the payroll department, so technically I was working two jobs. As a result, I was in the shipping room by myself for the majority of the time i worked there. The IT department was in a separate office next door to the main office that I was located, so that officially made me the guy that would be the communicative glue that would link the main office with the IT office, since I knew more regarding what was going on in the main office, I could report to the IT department in our meetings and let them know what was taking place. Considering more than half of the people in the main office became friendly with me and knew where I was located, they would just run to me rather than put in an email.

Regarding that dumb-ass policy, it was set for 2 reasons:

1. Corporate had a habit of checking (i.e. SPYING) on IT and their work, especially the amount of emails that would come in for support. The feeling I got was that they were thinking about axing one or more of us if the requests and work were low, so having the emails in the IT inbox would be proof that they would need to keep everyone in IT
2. Like a year after I started working for the company, IT had finally decided to have an actual ticketing system, and their decision was to use DeskPro. The idea was to create a separate help desk webpage specifically for internal use where people would go to the website, put in a ticket, and the ticket would go to the DeskPro system. They needed people to do this in order to test to see how the system would run and the walk-ins everyone in the main office were doing with me were obviously not helping in that regard.

 

[Never Scored]

I generally look after IT for the company I work for, but we're extremely small so it only takes up an hour or two of my time a week. The other day, owner wants me to go over to his house to see why the range extender he picked up wasn't working. I go over there he has one of those little extenders that plug directly into an outlet plugged in an outlet about a foot away from the router, with an ethernet cable going from the lan port on the extender to one of the lan ports on the router. I have no idea how he thought it was supposed to work.

 

[ducktales4gameboy]

While janitoring I ran into a cable modem setup where the client had paid some exorbitant amount for a 'cable booster'.


The left and center outputs were connected. Why.

There's also the perpetual joy of running into what I call the 'dueling banjos' setup where multiple machines on the same network are trying to act as DHCP (or tftp boot) servers, especially if the machines are in different fucking buildings and I have to walk back and forth half a dozen times to mess with cabling because both of them are Kensington locked to the walls and I haven't been provided the keys.

6TB of anime on our academy NAS.

This happened when I was in college except it was 2TB. This was in the late 2000s.

The sysadmin gave them some form of limited sudo privileges and told them they had a week to tag it, sort it and set up proper umasking/user accounts through samba for read-only access. Basically everything you'd require for an actual media asset management system in a professional setting.

They did not succeed, but the cause of failure was a squabble over something involving the filing heirachy rather than lack of effort.

 

[Some JERK]

You dont deal with enough manufacturing plants. SCADA is expensive to upgrade so i once had to support a two domain network so their C&C machines had an NT4 domain controller to talk to.

The NT4 and 2000 servers were baremetal with no backups

Industrial machines can be hilariously old, but nothing beats the good old US military. Popular media has people believing that all military tech is bleeding edge. That's only in-theater equipment and even then only if you want to kill someone. I've seen state-side military tech where there's no way that anyone who designed or installed it is still alive.

 

[White Devil]

Industrial machines can be hilariously old, but nothing beats the good old US military. Popular media has people believing that all military tech is bleeding edge. That's only in-theater equipment and even then only if you want to kill someone. I've seen state-side military tech where there's no way that anyone who designed or installed it is still alive.

DoD is still using fucking DOS.

 

[Smaug's Smokey Hole]

DoD is still using fucking DOS.

Wasn't Intel making the 486 up until recently for NASA at that thicc bucatini node? People always laugh at first when they're told about cosmic radiation.

And DOS is relatively recent, stack emulation on top of emulation on top of emulation of systems and you're building the computer equivalent of the tomb-throne of the emperor of mankind. My most alarming memory of this was working on an old ass IBM system that had moved on from the 70's using IBM hardware until the whole thing was eventually running in an emulator on DEC Alpha, the emulator was later emulated on Itanium because Digital went under and it was easier to just do it that way, I guess.