Category: Linux

New toy: Raspberry Pi 4

So of course, the first thing I want to do (after installing Raspbian and applying updates) was to add some aliases and switch the editor and history search commands.

Changing the editor to vim

Changing the editor to vim was easier, once I knew how.

  • Install vim
  • update-alternatives

Vim does not come installed by default on a Raspbian. But adding it is easy enough:

sudo apt-get install vim

I had found someone that said (when thing weren’t working the way I wanted) to sudo apt-get install vim-full ← this does not work! There is no package “vim-full” from which to install. At least not on my fresh-out-of-the-box Raspbian machine.

sudo update-alternatives --config editor

This opened up a list of available editors, numbered for selection, with an asterisk next to the default. It was set to nano, but I wanted vim.

Except that there were two vim choices: vim.basic and vim.tiny

Which one to choose? Neither looks like vim.full to me. 😉

Turns out I wanted vim.basic

Now, I can be in the less utility, and if I want to edit the file (and I already have permission to do so) I can hit the v key and be editing in vim.

Adding some aliases

This was super easy. I created a file, ~/.bash_aliases and added the alias commands to it.

alias ..='cd ..'
alias ll='ls -l'

Changing the history search keystrokes

This one was the closest to what is described in my How to make Ubuntu have a nice bash shell like OpenSuSE post.

sudo vim /etc/inputrc

Find the commented out commands, and uncomment them:

# alternate mappings for "page up" and "page down" to search the history
# "\e[5~": history-search-backward
# "\e[6~": history-search-forward

All I have to do is to delete the “#” character that declares \e[5~ and \e[6~ to be a comment. With vim, this is the x key

rsync is wonderful, but ….

rsync /datastore/61/E4 /newserver/61/E4

is wrong and will mess you up!

Imagine if you will, that you have a whole bunch of data stored on an old server, and you need to copy it to a new server. The rsync utility would be an obvious way to go. There are things about the job and rsync that you might want to tweak, though, and that’s where things get ugly. Part of this is bash’ fault.

Imagine if you will, that your data store is 120 million small files (emails) stored in 256**3 directories. 256 cubed is 16,777,216 sub directories.

The programmer that created the data store to hold all these files needed subdirectories to put the files in. Linux doesn’t really like 20,000+ files in one directory. It would be better to have more subdirectories, with less files per subdirectory. So the programmer started with a loop:

for 00 .. FF mkdir

Then the programmer did a change directory into each of those directories he just made, and did the exact same thing. 

cd 00;for 00 .. FF mkdir;cd ..
cd 01;for 00 .. FF mkdir;cd ..
cd 02;for 00 .. FF mkdir;cd ..
...
cd FF;for 00 .. FF mkdir;cd ..

That gets you to 256 squared, which is 65,536

And then the programmer did a change directory into each of those directories he just made, and did the exact same thing. All 65,536 second level subdirectories got a third level of another 256 subdirectories. That gets you to 16,777,216 which is 256 cubed.

So your file server directory structure might contain this:

/datastore/61/E4/7D

Inside good old 61/E4/7D there might be twenty to thirty files, each one holding the content of an email, or a metadata file about the email. The programmer was pretty good about filling all of the datastore subdirectories to nineteen files each, then twenty files each, then twenty one files each. No Linux system is going to have a problem with twenty one files in a subdirectory.

The only real problem here is if you need to traverse everything in /datastore – this takes forever

Back to the problem of copying everything from /datastore to /newserver. Let’s assume that /newserver in on a different machine, and we are using remote file system mount command to make the remote machine appear to be a local disk (mount point).

You might think the rsync command ought to look like this:

rsync --archive /datastore /newserver

There are two things that make this sub-optimal. First, it is single-threaded. Second, there is no progress feedback.

The single threaded part isn’t so bad; it just means that we are losing speed due to rsync overhead. The server has twelve cores, the network is 10 Gbps Fibre Channel, the /datastore disk has multiple spindles, but rsync was designed for slow networks way back when in the bad old days.

At this point, you might ask “why not do a straight cp -r” (copy command, recursive)? It’s not a terrible idea; but, what if there were a network glitch? The entire cp -r would have to be started over, and every bit already copied would be copied again. This is where rsync shines: if the files in the destination are the same as the source, the copy is skipped. cp -r also suffers from the same lack of progress feedback.

Did I mention that the 120 million files are also 9.3 terabytes of files? I really don’t want to get to 98% done and then have a network glitch cause me to copy another 9.3 TB over, which would be the case with cp -r

The tests I’ve done indicate that four rsync commands, running simultaneously, copied the most data in the shortest period of time in my environment*. More than four rsync commands at once, and I started to saturate the disk channel. Less than four rsync commands, and something is waiting around, twiddling it’s thumbs, waiting for rsync to get busy with the copying again, which it will do, as soon as it finishes up with the overhead it’s working on.

The other problem is a lack of progress feedback. The copy is going to take multiple days. It would be nice to know if we are at 8% complete or 41% complete or 93% complete. It would be nice to be able to compute what the percentage complete is.

Well, how about 64K rsync commands, each with a print statement of the directory it is processing? And if we could run four of them in parallel, we could get the multiple jobs speedup too.

You might think the rsync commands ought to look like this:

rsync --archive /datastore/00/00 /newserver/00/00
rsync --archive /datastore/00/01 /newserver/00/01
rsync --archive /datastore/00/02 /newserver/00/02
rsync --archive /datastore/00/03 /newserver/00/03
rsync --archive /datastore/00/04 /newserver/00/04
...
rsync --archive /datastore/FF/FF /newserver/FF/FF

but WOW would you ever be wrong!

Remember old /datastore/61/E4/7D up there? This format for rsync would put E4 in the source under E4 in the destination! In other words, although the source looks like this: /datastore/61/E4/7D the destination would look like this: /newserver/61/E4/E4/7D

To be done right, the command needs to look like this:

rsync --archive /datastore/00/00/* /newserver/00/00/
rsync --archive /datastore/00/01/* /newserver/00/01/
rsync --archive /datastore/00/02/* /newserver/00/02/
rsync --archive /datastore/00/03/* /newserver/00/03/
rsync --archive /datastore/00/04/* /newserver/00/04/
...
rsync --archive /datastore/FF/FF/* /newserver/FF/FF/

The source needs a trailing slash and asterisk to tell rsync to copy the stuff underneath the source (not the source itself) to the destination (which is finished with a slash).

Enter the problem where bash is a pain in the ass.

Well, before I go there, let me mention that it wasn’t too bad to write a Perl script to write this bash script, and do three things per source and destination pair:

echo "rsync --archive /datastore/00/00/* /newserver/00/00/"
rsync --archive /datastore/00/00/* /newserver/00/00/
echo "/newserver/00/00/" > /tmp/tracking_report_file

The first line prints the current status to the screen. The second line launches the rsync. The third line overwrites a file, tracking_report_file, with the last rsync finished.

So, crank up screen first, launch the bash script, and some number of days from now, the copying will be done.

That /tmp/tracking_report_file gives me a pair of hexadecimal pairs, which I can then use to compute percentage complete. For example, when /newserver/7F/FF updates to /newserver/80/00, then we are going to be just over 50% done.

Heck, I can detach from screen, and I don’t even have to watch the rsyncs happen. I mean that I do need to, but I don’t have to. Better yet, I can take the same routine that converts the pair of hexadecimal pairs into percentage complete and wrap that inside a cron job that sends an email. Progress status tracking accomplished!

But this does not solve the single-threaded rsync problem.

And ultimately, I could not get it done.

What looked to be an okay solution was using the find command, to feed into xargs which could do shell stuff in parallel. I even got as far as getting bash shell variables to create the rsync --archive /datastore/00/00/00 /newserver/00/00/00 part.

Okay, that would be 16 million smaller rsyncs instead of 64 thousand larger ones, but I might even be able to bump up the parallelism to six or eight or nine.

But the serious problem the rsync –archive /datastore/00/00/00 /newserver/00/00/00 command has, is the naive problem: the missing trailing slash and asterisk are going to put the source underneath a destination. I need to put the trailing slash and asterisk on there.

And bash says “that’s a nope”

Trailing slashes and asterisks are automatically culled from output, because (reasons).

Oh well. The find command also spits out the directories it finds in rather random order. My bash script with sequential rsyncs by sorted order means that the last one complete really is some-percentage-of-the-total done. But if find chooses to spit out /datastore/b3/8e/76 instead of /datastore/00/00/00 then my status tracking doesn’t actually work. I would be forced to traverse all of /newserver/ and count which of the 17 million are complete; which would take freaking forever.

Yes, I said 17 million. Did you notice that the programmer that created subdirectories did some of them in lowercase hexadecimal? That happened when we brought in another email system (Exchange). Lovely.

*the last time I did this migration, although it was on a four core box, then.

Update to AMD Ryzen 1700 and power sleep failure

I had written about a fix for my machine because it has a slightly older AMD Ryzen 1700 CPU. I recently re-installed an older version of OpenSUSE (wiping out the previous OS drive contents and replacing it). This did what I wanted it to do; but, it also wiped out the fix for the power sleep problem. I went back, and tried to implement the same fix, but it didn’t work. So here’s my note about a better fix.

I still am using the script in /etc/init.d which I named set_c6_acpi_state_disabled.sh

I did have to edit it to invoke python3 instead of just python.

#!/bin/sh 
# ScriptName=set_c6_acpi_state_disabled 
/usr/bin/python3 /home/blah/zenstates.py --c6-disable

zenstates.py can be found here.

But instead of messing with symbolic links to script files in places, I’m just adding a crontab entry to the root user:

@reboot /etc/init.d/set_c6_acpi_state_disabled.sh

OpenSUSE 15.1 → 15.2 upgrade; with nVidia it’s a disaster due to nouveau

Upshot is, the installer has two different options depending on if you are in graphical mode (init 5) versus text mode (init 3). Graphical mode (which was not recommended) at least had the possibility of working.

Back story: at work more than a decade ago, the whole team got new computers, but one of my co-workers preferred his MacBook laptop. After his PC sat unused for a couple years, I got the idea to ask him if he minded me having his dual monitors. He was okay with that. Of course, to drive those two monitors, one needs a dual monitor video card, so I took that too.

Now my machine had two identical video cards and four identical monitors. It was pretty sweet. The nVidia drivers had handled that just fine. Well, until ….

Then some upgrade happened (in 2017), and suddenly the nVidia driver could only light up one of the two video cards. I really liked having four screens, so I went and bought a single nVidia card with four outputs – NVIDIA Quadro P600. Life was good.

However, with this configuration, all the upgrades to OpenSUSE since then are just miserable. This weekend, I tried again. Awful, as usual.

The problem is that the install wants to replace the nVidia (proprietary) driver with the nouveau (open source) driver, but that never works right.

The other, bigger, problem is that if (in text mode) you don’t agree to let nouveau foul over your machine, you don’t get to upgrade at all.

I’ll try to illustrate the steps:

In text only mode: zypper dup, and at the end I was warned that the nouveau driver was going to be used. The prompt says “do you agree? yes/no: “

If you type “no”, the upgrade stops.

If you type “yes”, the upgrade continues and then trashes the graphical mode configuration. It installs the nouveau driver which doesn’t work with a single card and four monitors.

After banging my head against the problem for too long, it gets time to give up, download the ISO, burn a DVD, boot off the DVD, and choose “install” – completely wiping out all sorts of needed stuff. 🙁

But doing that (a fresh install), I get a graphical install. I get to the point where one of the prompts warned that the nouveau driver was going to be used. The prompt says “do you agree?” There are two buttons: “I agree” and “I disagree”. But most importantly, there is a checkbox for “Install using software emulation only”

That checkbox disables nouveau from breaking the system. Well, it disables nouveau completely, which prevents the system from being broken by nouveau. During the install, I saw a log line that nouveau was being locked from zypper updates, and that a blacklist entry was being added for nouveau to prevent start up.

Finally, I had a system that would boot. It wasn’t particularly stable, but at least it could boot and present the graphical login prompt. The graphical session wasn’t stable or working well, but at least it wasn’t booting to a single blinking text cursor in the corner of one monitor.

It turned out it was easiest to ssh in from a different machine, run the zypper command to add the nVidia repository, and install the nVidia proprietary drivers.

One more reboot later, and I’ve got a stable, four monitor, graphical environment once again. Hooray! 😀

One thing a person should always to with Linux: use a separate hard drive for /home

This way, installing a new operating system (whether it is Mint, OpenSUSE, Kubuntu) does not mess with your files. The new install sticks to the Operating System disk, and doesn’t mess with your /home disk.

So, all my VM guests are fine, my Perl scripts are fine, my login environment (KDE) is fine. That’s all good.

What’s bad about replacing the OS because the upgrade trashed the system is that my personal crontab file is not stored under /home.

I didn’t remember to go find and back up that file, so now it’s gone. I’m going to have to re-invent it from memory. All the scripts I write output log files; but, those get written to /var/tmp – which also gets wiped out when a fresh install happens.

Although I had printed the names of all the repositories I use (and put that file on /home) I was in a hurry and didn’t realize that the screen didn’t list the URLs. Whoops.

And of course, since the way out of the broken system was a fresh install, all the Perl modules I’ve installed in the last few years need to be discovered and installed again.

But, I’m running OpenSUSE Leap 15.2 now. Leap 15.1 is planned to go end-of-life next month, so I thought it about time to get to 15.2. That’s done, but man was it painful.

Lightsail + Bitnami has changed the underlying image – I get to learn debian now :-)

So far, it’s been easy. I have a post about How to make Ubuntu have a nice bash shell like OpenSuSE and it barely needs any update.

But it does, a little.

sudo update-alternatives --config editor

In the menu that was presented to me, Option 2: 2 /usr/bin/vim.basic was the right one. There’s nothing there to tell me the difference between vim.basic versus vim.tiny (I wanted whatever vim.fullsuperduperthrowinallthebellsandwhistles is) – but vim.tiny was not it.

touch ~/.bash_aliases
vim ~/.bash_aliases
(insert mode)
alias ll="ls -la"
alias ..="cd .."
(escape)
:wq

and lastly, the step that is the same:

sudo vim /etc/inputrc

Find my way down to where the comments say alternate mappings for “page up” and “page down” to search the history and then hit the x key a few times to delete the characters which comment out the functionality I want.

Log out, and log back in. It’s good. 🙂

Wow Perl PAR::Packer sucks

I presented a problem to my boss, where I need to let about 30 desktop technicians run some code I’ve written. My boss said when he is in that position, he writes it in PowerShell, and uses PS2EXE. This is good idea. I found that in the Perl world, the same idea is in PAR::Packer.

A super simple script, running on my Linux box, takes less than one second to run. Open up an SQLite database, fetch all 20 records, sort them, print them. Simple. Less than one second.

The Windows .exe version takes 19 seconds.

Every. single. run.

There is no way that the on-call technicians who have to run the scripts I’m writing are going to be happy with that. And it would make me feel bad, inflicting that sort of this-is-time-of-my-life-that-I-am-not-getting-back on some poor soul who got a call at two in the morning, to deal with whomever for whatever.

Normally, on Windows, I used WinBatch. Did so for 20 years. But alas, WinBatch was always a for-pay product, and eventually hobbyists wrote AutoHotKey (or was it Auto-It?) (for free) to do everything WinBatch does. Also, the real goal was to take my Perl scripts that use REST to get and put JSON or XML at a web service. WinBatch has lots of old extenders, but rarely any new ones. I don’t know of a REST extender for WinBatch. I don’t know of anything in WinBatch that does what DBI does in the Perl world. The WinBatch answer is to install MS Access (or whatever) and use COM to drive the actual database client. Avoiding installing software is part of the goal here. I need something That Just Works.

And preferably works in a second or two; not twenty.

I’ve got REST modules written in Perl. I’ve gotten far enough in my Perl skills to put standardized code in modules, and then use those modules. But that meant putting modifying the search environment for finding modules. Guess what doesn’t work with PAR::Packer?

Okay, crap, I can move all my modules into the root directory where my scripts are. This is stupid, but I’ll do it.

And then I find that the execution speed of even a simple script is terrible.

Searching for a solution, of course the answer is “You’re doing Perl wrong, if you don’t want to install an entire development environment on every workstation”.

Well, thanks for nothing. Apparently I need to give up Perl and start learning PowerShell. PowerShell does rather look like Perl, so maybe it won’t be too terrible of a transition.

Bitnami phpmyadmin

Just a quick note for me to easily find and remember how to access PHP My Admin on a Bitnami WordPress instance

From the command line on my local machine:

ssh -4 -N -L 8888:www.gerisch.org:443 -i $insertpathtopemfilehere nottheadmin@gerisch.org

And then in a browser:

https://www.gerisch.org:8888/phpmyadmin

Lastly, remember that the login name to phpmyadmin is root (not the Bitnami application password, or any other user name).

Because public Internet access to PHP My Admin would be a Very Bad Idea, the Bitnami WordPress image is configured such that PHP My Admin refuses to run, if the requests don’t come through www.gerisch.org

This is a good idea.

But what that also means is that I need something listening on my www.gerisch.org address, that can forward the network traffic to the remote web server.

ssh -4 says use IP v4 addresses only (suppresses IP v6 errors if your machine doesn’t have that).

ssh -N says do not execute remote commands (all we’re going to be doing here is port forwarding).

ssh -L says local to remote port forwarding will be done.

8888:www.gerisch.org:443 says the local port to listen on is port 8888, the local address to listen on is the home address of www.gerisch.org, and when listening on the “server” www.gerisch.org, know that it will be listening for port 443 traffic (https instead of http). Another way of thinking about this is that your web browser that is throwing HTTP GETs and PUTs will be throwing them at port 8888, since that is the port the service is listening on. But when the traffic is thrown across the Internet, ssh is going to throw the traffic to www.gerisch.org port 443. Yet, www.gerisch.org:443 is really just a front for gerisch.org:443

ssh -i says to use a public/private key pair for logging in (instead of a password). $insertpathtopemfilehere is the variable that holds the path to the .pem file.

ssh nottheadmin@gerisch.org is the actual remote login name and server name.

More power sleep Linux stuff – or how to set permissions to make a keystroke out of it

This command does successfully put the computer to sleep (and thus the screens eventually go dark and no longer light up the room like stadium lighting):

echo freeze > /sys/power/state

But mere mortals don’t have permission to do that.

I added a script in /etc/init.d which does a chmod 666 /sys/power/state on startup (see the previous post about editing /etc/init.d/after.local)

Credit where credit is due: a gentleman named Aaron Ball posted this at his web site oper.io – clever logo, too, to combine the power switch icon as “.io” – but I digress.

Then another tiny script that simply does the echo command for me:

#!/bin/sh
/usr/bin/echo freeze > /sys/power/state

I had started to go down the road of a sudoers file entry, which would give anyone permission to run this script; but the problem isn’t who runs the command. The problem is that the target of the echo command, /sys/power/state, isn’t going to allow writing by a script (even if run by sudo).

Last step was to add a keystroke to my KDE shortcuts. System Settings –> Custom Shortcuts –> Edit –> New –> Global Shortcuts

Name the action, assign the keystrokes to invoke it, and make the action the tiny script above. Works like a charm. 🙂

AMD Ryzen 1700 and power sleep failure

I had bought all the parts for a new system at the end of 2017, and was mostly happy with it. The motherboard was an AS Rock Taichi and the CPU was an AMD Ryzen 1700. I bought the highest MHz AMD Ryzen I could get except I avoided the 220 Watts TDP; power draw on this chip is about 65 Watts TDP.

Certainly, I was thrilled with system performance, and very reasonable price. But sleep states were a problem. The system would go so deep into sleep, it would never wake up. I would have to press and hold the power switch to get it back: not good. Once in a while, the box would freeze hard too; usually while scrolling a Facebook page. So I had this sense of unease that I’d made a mistake buying the AS Rock Taichi X370 and trying to run Linux on it.

This last birthday, I bought myself a replacement motherboard: an MSI X470 Gaming Plus. I spent my birthday pulling out the Taichi motherboard and putting in the 470GP.

And my ACPI sleep problems did not go away. Rats!

I’ve taken some vacation time, and looking through the log files, I did find an error message that lead me to some vital information. The AMD Ryzen 1700 has an ACPI sleep state – C6 – which Linux doesn’t play nice with. A patch was offered to the Linux maintainers, but not accepted. I don’t know why, and I’m not sure it matters, either.

But what someone did, is make a Python script that pokes and prods the correct bits in the Ryzen 1700 to have it declare that ACPI sleep state C6 is not available / should not be used.

I haven’t had a sleep state problem with this rig since. 😀

First, I needed this: Github ZenStates. I saved this script as /home/myhomelocation/zenstates.py

Then I needed two bash scripts to run it after system startup:

/etc/init.d/after.local

which contains (and is executable): 

#!/bin/sh
/etc/init.d/set_c6_acpi_state_disabled.sh

/etc/init.d/set_c6_acpi_state_disabled.sh

which contains (and is executable):

#!/bin/sh
# ScriptName=set_c6_acpi_state_disabled
/usr/bin/python /home/myhomelocation/zenstates.py --c6-disable

In theory, this could be a single script: after.local could be the script to run the Python script zenstates.py. But what if I find I want more than one script to run after system startup? This way, I just add another line to after.local

This week I talked with my brother, who found a relevant piece of information (while researching something else). Apparently AMD will give me a new CPU that doesn’t have this problem, as a warranty repair. So I guess this is AMD being honorable enough to admit they made a mistake here; that is nice. The caveat is, they cannot trust people to not fry their CPU and use this as an excuse to get a replacement under warranty. So the drill becomes: get an RMA from AMD, remove the Ryzen 1700, ship it (at my cost) to them, they test the CPU to make sure it’s not fried (is otherwise good except for the C6 power state problem), and then they ship a replacement. Total turnaround time is probably one month.

Do I want this, my main system, to be down for a month until the AMD CPU RMA SOP EOP FTW? The warranty on the CPU is three years, and I bought it two and a quarter ago.

I think I’m good, with just a software patch. I’m just happy that r4m0n found and supplied a patch.

Thank you r4m0n. 🙂

New OpenSuSE installation – Facebook videos / GIFs don't play

This is just a reminder to myself that when I install fresh OpenSuSE, that when Facebook videos don’t play (but Youtube does), the solution is to go into software management, and to add libxmp4 and MP4Tools. I think that was it. It was important to allow vendor change, and I had to do a lot of acknowledgments for that.

I see that I changed:

  • libxmp4
  • libwx lots of stuff
  • bento4
  • MP4Tools
  • ffmpeg-4

Previously, I had added the Adobe Flash stuff, and that had fixed some of the trouble, but not all of it.