f-log

OMG! I have found another un-blogged post from Oct 2015. Looks like it was busy couple of months, including the life changing leg break. So I am letting myself off.

As it is perfectly relevant now as it was then I have reproduced in full.

As part of my new job I shuffle files around servers. Normally this is quite simple and straight forward but one Windows 2008 R2 server decided to be awkward and not allow me to remotely connect.
There was no error message or any other traceable problem so I just setup a share on the c:\inetpub folder which holds the web site files. I set the share only to myself and got on with my job.
The following day I was off with one of the kids as they could not go to school with an infectious illness. I went in the next day to find all hell had broken loose! It transpired that when I set the permission on that share it removed all the carefully curated local accounts that were used by the web server. The web server had been unable to read the files it need to serve and I got a chewing out.
Perplexed by this turn of events and the fact no body seemed to think this was expected behaviour I tried some experiments(on non essential folders) and was able to repeat the issue.
Due to this repeatably I assumed it was one of those secret lore things and there would be an obscure reference to it in the official documentation or a few blogs discussing it. But no, I could only find one old Stack Overflow question that had never been properly resolved.
So I created my own and tried a few more experiments, different machines different windows and IIS versions. Some times it had the issue other times it did not. I decided the only sensible thing was not to use Windows shares in that way again, consider them tainted!

Ok, less than an hour to go until 2018 and I found this gem from Oct 2015.

As it is perfectly relevant now as it was then I have reproduced in full.

I want to say I have been back and completed my Pebble Time Magic 8 ball app, but I have not had the time. What I have done with it though is to take it on holiday to sunny Spain where it more than happily weathered the sun, sea and sun cream.
So impressed with this little delight. Especially not having to worry about getting my phone out to read text messages and being able to reject calls.
But the most amazing thing has to be the battery life, 7 days almost on the dot. Even when it is on 0 power and stops functioning as a smart watch it switches to a simple digital time face and continues to show the time!

It is just a few hours before 2018 and I just needed one more post to try out some mobile changes...

Just knocked up some very simple previous and next buttons in Inkscape to make switching pages easier.

Do you remember my Pi security camera made out of a toilet roll?

Well I do and it had been working fine in the past, so I was surprised to find it non operational when I tried to use it over the holidays.

I tried to ssh to it but it either dropped the connection before I logged in or shortly after. It became apparent that this Pi was continuous rebooting.

I checked the log files and sure enough the main script was failing and causing a reboot. The launcher shell script simply starts the Python code and runs in a loop, if the Python code stops the next line in the shell script is reboot.

2017/08/06 21:17:26 rebooting on exit of launcher

That instance was because api-content.dropbox.com was not available. Not sure if dropbox was down or my connection to it. Any way, the script found the issue rebooted and ran fine by time it came up again.

Then
dropbox.rest.ErrorResponse: [500] u'The server has either erred or is incapable of performing the requested operation.'
2017/08/16 22:04:56 rebooting on exit of launcher

Dropbox again, then on 28th August

File "/usr/local/lib/python2.7/dist-packages/dropbox/rest.py", line 227, in request
    raise ErrorResponse(r, r.read())
dropbox.rest.ErrorResponse: [400] u'v1_retired'
2017/09/28 19:31:23 rebooting on exit of launcher

and from then on it just rebooted, loaded, failed and rebooted again, and again and again.

Seems Dropbox have dropped support for the v1 script API I was using.

"Catastrophic system failure is a good thing ™" is a t-shirt I used to wear to work, including when I worked at Microsoft. It simply means that when something goes horribly horribly wrong, it is an opportunity to make better. The ™ was a jibe at how many corporation used the "it's feature not a bug" excuse.

So what we have here is number of failures.

1: I did not heed Dropbox's warning. They did email me one, so I wonder if I checked the code at that time and it still worked. But their plan was shut it down gradually.
2: I was not monitoring this Pi. To be fair, I did have a plan to do a big monitoring solution... but never got around to it.
3: The reboot on script exit was flaky to say the least. Although it did work for a couple of issues it was always headed in this direction. Could have at least had a sliding timeout so if it had been going on for a while I could have ssh'd in.
4: As well as monitoring there were no alerts. Could of sent an email or any other number of "Help" messages.

To fix;
Disable the reboot line in the script. Except I cannot save the change, as all the log files have filled up the tiny 4GB SD card. Or not, there was just under 1GB of captured photos. they only get removed once successful uploaded. and /opt/Wolfram was 500MB.

Maybe simply updating the packages will fix the issue...

sudo apt-get update
sudo apt-get upgrade -y


But afterwards, my short test script still erred with

dropbox.rest.ErrorResponse: [400] u'v1_retired'


and

sudo pip install --upgrade dropbox


Cannot fetch index base URL http://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement dropbox in /usr/local/lib/python2.7/dist-packages
No distributions at all found for dropbox in /usr/local/lib/python2.7/dist-packages


Hmmm,
sudo easy_install dropbox
Searching for dropbox
Best match: dropbox 2.2.0
Adding dropbox 2.2.0 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages
Processing dependencies for dropbox
Finished processing dependencies for dropbox


sudo pip install dropbox
Requirement already satisfied (use --upgrade to upgrade): dropbox in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): urllib3 in /usr/local/lib/python2.7/dist-packages (from dropbox)
Cleaning up...

sudo pip install --upgrade dropbox
Cannot fetch index base URL http://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement dropbox in /usr/local/lib/python2.7/dist-packages
No distributions at all found for dropbox in /usr/local/lib/python2.7/dist-packages
Storing complete log in /root/.pip/pip.log


might be related to this issue
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1658844

wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py --prefix=/usr/local/
sudo pip install --upgrade dropbox

new error, looks like the API has not got the same objects :(
Traceback (most recent call last):
File "dropbox_test.py", line 5, in <module>
    client = dropbox.client.DropboxClient('DEADBEEFDEADBEEFDEADBEEFDEADBEEFDEADBEEFDEADBEEFDEADBEEFDEADBEEF')
AttributeError: 'module' object has no attribute 'client'


https://github.com/dropbox/dropbox-sdk-python

So changing dropbox.Client to dropbox.Dropbox resulted in needing to change

client.account_info()
to
client.users_get_current_account()

client.put_file(path,file)
to
client.files_upload(file,path)

but that fails with

TypeError: expected request_binary as binary type, got <type 'file'>

so that became two lines
data =f.read()
client.files_upload(data,path)

then

f, metadata = client.get_file_and_metadata(frompath)
changes to
metadata, f = client.files_download(frompath)

and then the code to save the file was

out = open('magnum-opux.txt', 'wb')
out.write(f.read())
out.close()

becomes

out = open('magnum-opux.txt', 'wb')
out.write(f.content)
out.close()

but it does work.

there was also a test I did to get the root folder metadata
folder_metadata = client.metadata('/')
became
folder_metadata = client.files_get_metadata('/')

but then get error

Error in call to API function "files/get_metadata": request body: path: The root folder is unsupported.

changing it to my test folder did work.

Then there was the missing forward slash. '20171229/20171229224356_1.jpg' used to work with the v1 API but now gives an error
dropbox.stone_validators.ValidationError: '20171229/20171229224356_1.jpg' did not match pattern '(/(.|[\r\n])*)|(ns:[0-9]+(/.*)?)|(id:.*)'

Prepending the forward slash fixed that and now all my code is v2 compliant and is tested to work.

FYI the v2 API was green lighted in 2015 and the V1 API depreciated 2016
https://blogs.dropbox.com/developers/2016/06/api-v1-deprecated/
and turned off in 2017
https://blogs.dropbox.com/developers/2017/06/updated-api-v1-deprecation-timeline/
migration guide
https://www.dropbox.com/developers/reference/migration-guide

Must have created my v1 based script just before they announced v2 in 2015 and or the end date of 2016/2017 must have seemed a world away.

You know if I have to post to the Gentoo forums then things have gone bad.

Every time I "get mail" in Claws mail, the status bar shows filtering to spamassassin for 7-8 minutes. But then I get any new mails correctly.

To try and narrow down the problem I ran

egrep "@" /var/log/messages | egrep -o "spamd:.+$" | sort | uniq -c | sort -n    
which showed it was the same messages each time. 198 of them. I then checked the POP3 mailbox that my mail is picked up from, but there were no messages with those email addresses.

Enabled "Show receive dialogue" and Claws clearly says "no new messages" but the status line reads "Spamassassin Filtering message n/198" where n = 0-198.

When there are new messages Claws returns almost immediately, correctly reporting the number of new "real" messages and then the "Spamassassin Filtering message n/N" where N = 198+new.

/var/log/messages spamd entries for these 198 all appear to have spammy email domains.
e.g.

Code:    
Dec 26 22:33:12 MACHINE spamd[22749]: prefork: child states: II
Dec 26 22:33:13 MACHINE spamd[24297]: spamd: connection from localhost [127.0.0.1]:56708 to port 783, fd 6
Dec 26 22:33:13 MACHINE spamd[24297]: spamd: setuid to MYUSER succeeded
Dec 26 22:33:13 MACHINE spamd[24297]: spamd: checking message <c332257553e5d5f138a77a9e350c908b@alrowadms.com> for MYUSER:1000
Dec 26 22:33:14 MACHINE spamd[24297]: spamd: clean message (4.6/5.0) for MYUSER:1000 in 0.4 seconds, 4651 bytes.
Dec 26 22:33:14 MACHINE spamd[24297]: spamd: result: . 4 - BAYES_99,BAYES_999,DKIM_SIGNED,HELO_MISC_IP,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RDNS_NONE,T_DKIM_INVALID scantime=0.4,size=4651,user=MYUSER,uid=1000,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=56708,mid=<c332257553e5d5f138a77a9e350c908b@alrowadms.com>,bayes=0.999765,autolearn=no autolearn_force=no
Dec 26 22:33:14 MACHINE spamd[22749]: prefork: child states: II
Dec 26 22:33:15 MACHINE spamd[24297]: spamd: connection from localhost [127.0.0.1]:56710 to port 783, fd 6
Dec 26 22:33:15 MACHINE spamd[24297]: spamd: setuid to MYUSER succeeded
Dec 26 22:33:15 MACHINE spamd[24297]: spamd: checking message <c10e2276c9c00d28d61220e031890465@bricksitalia.com> for MYUSER:1000
Dec 26 22:33:16 MACHINE spamd[24297]: spamd: clean message (4.5/5.0) for MYUSER:1000 in 1.2 seconds, 3142 bytes.
Dec 26 22:33:16 MACHINE spamd[24297]: spamd: result: . 4 - BAYES_99,BAYES_999,HELO_MISC_IP,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RDNS_NONE scantime=1.2,size=3142,user=MYUSER,uid=1000,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=56710,mid=<c10e2276c9c00d28d61220e031890465@bricksitalia.com>,bayes=1.000000,autolearn=no autolearn_force=no


Found an error in syslog "Dec 26 21:09:13 MACHINE spamd[22192]: spamd: failed to create readable default_prefs: /home/MYUSER/.spamassassin/user_prefs"
That I was able to fix by editing /etc/conf.d/spamd and setting SPAMD_RUN_AS_ROOT=true. That fixed that error but the behaviour continues. As part of that issue I also deleted the /home/MYUSER/.spamassassin/user_prefs file, it has been automatically recreated.

So, on each collection of mail, Claws appears to only get new messages and invokes spamassassin, but for some reason spam assassin tries to process 198 "other" messages.

Where are these messages and how can I stop spamassassin processing them ?

...

And then things turn nasty.

I shut down claws and when I reopened it the same sequence of events start to take place and then it crashed ...

suspecting a corrupt file in UTF8 conversion

running in debug mode

claws-mail --debug

spamassassin.c:214:Filtering message 3566
procheader.c:155:generic_get_one_field: empty line
procheader.c:155:generic_get_one_field: empty line
procheader.c:155:generic_get_one_field: empty line
tnef_parse.c:295:Tnef parser parsing part (42094).
tnef_parse.c:297:content: /home/MYUSER/.claws-mail/mimetmp/claws.5BDZBZ
Attempting to parse /home/MYUSER/.claws-mail/mimetmp/claws.5BDZBZ...
suspecting a corrupt file in UTF8 conversion

Tried removing the claws.5BDZBZ file but on the next run it was recreated with a different name.

Found a single mention of the same issue https://groups.google.com/forum/#!topic/linux.debian.bugs.dist/rp-XIynhoMU which led me to re-emerge ytnef. Which was pointless because the ytnef claws-mail uses is built in.

But I found if I started claws-mail in offline mode and then disabled the tnef plugin I could then restart in online mode and not crash out.

claws-mail --offline

Now I knew what folders to look in I started poking around for the 198 messages.

/home/MYUSER/.claws-mail/tempfolder/processing-1
Contains 201 files, of which 3 were not email messages, BINGO 198. The date range was surprising

2016-11-04 21:11:15 - 2017-02-24 06:00:00

I knew this issue had been going on a while, but as I always ended up with my emails, I just got used to doing something else while waiting.

So I renamed the /home/MYUSER/.claws-mail/tempfolder/processing-1

claws recreated the folder and ... IT WORKED!

Re-enabled the "tnef" plugin and it continued to work.

I had a look in the renamed processing folder and found each message was indeed spam and curiously they were all to a single email address. Which is odd because I get emails from all sorts of addresses forwarded to this mailbox.

Probably should have looked at fixing this issue before, like 14 months ago :)

Wow! I have really blasted through the last 20+ components.



All squares are now covered, meaning a 3D model exists for all those components that I could identify. The Yellow is where the texture needs cleaning up to remove shadows from the original camera shots. The tiny bits of green are texture points that need tidying. For instance the big 4R7 octagonal components are visible with perspective, meaning you see the edge not the top, which in turn misrepresent their position on the board.

I knew the last components were going to be the most complicated, due to text and other designs. For some I just used the Blender built in font and it looks 80% the same and the for the others I traced the source images and added the results. To begin with I loaded the images as a background into Blender and manually added curves. This took a long time and seemed a bit much. then I remembered that Inkscape can trace bitmaps and save them as SVG. Although tiny, these SVGs then represented mathematical curves.

Before I could CTRL+J (join) the curves/text to the components I had to convert them to Mesh objects.



So, now I just need to paint out the shadows and the "texture tidying" before revisiting my material library in Blender and ...

Well lets just say I have a few fun ideas for when I have a finished Pi3 model :)

OK, the underside is complete (bar the shadow removal on the texture)




I will not tackle the image editing until the last of the components are modelled. I also expect to revisit a number of the materials when used with an HDRI for lighting.

It is important to me that I show that the Pi is progressing


As you can see, almost all the bottom components are complete. The Yellow smudges are where I need to remove a shadow from the original photography, now real 3D elements cast real shadows. Those shadows cast in different directions based on light sources :)

Pi is progressing


compare to last months Posts

So, I keep forgetting I own the GoPro Hero, but I took it out for "work" today. It snowed and it NEVER snows around here, kids went mad and we all built a massive snowman.

I set up the GoPro Hero on the windowsill and started a time-lapse. This being the cheapest of the GoPros there are a number of limitations.

Fused case; cannot take camera out of case.
Battery fixed; cannot remove battery, have to charge camera directly.
5mp at 2592x1944; square shots.
Max SD card size of 32gb (class 10).
All time-lapses are fixed at one frame per 0.5 seconds.
Time-lapses are stills not a video.

But as it is often as much as 1/4 of the price of other GoPros I can let all that slide and the waterproof case is good to 131ft. Not to mention it was a gift, so cost me nothing.

So I spent my time building the time-lapse stills into a movie and ... It is boring, really boring everything moves too slowly. So I got to wondering if there were any hacks to fix the frame rate.

https://github.com/KonradIT/autoexechack/tree/GoPro/HERO/TimeLapse

has a collection of script files that you just save to the root of your SD card and it just works!

Here is the 30 second per frame script.

sleep 1
t app key mode
sleep 15
t app key record
sleep 1
reboot yes


Looks like it switches modes waits 15 seconds takes a picture and then reboots to repeat it all over again.

15 seconds?? No idea how it works but a test recording of 5 mins of a clock showing the seconds, confirms that each still is taken at exactly 30 seconds. The other scripts at that location are for 10 seconds per frame, where the value is 5 and 5 seconds per frame where the value is 1.

It is worth pointing out at this point that after adding the script (with the special name "autoexec.ash"), turning on the camera will immediately start executing the program, which caught me unaware as the time-lapse screen still said 0.5 and I ended up recording my confused face.

The GIT repo has a ton of other hacks for all types of GoPro cameras.

https://github.com/KonradIT/autoexechack