Some questions and such that have arisen out of the "Re-inventing..." project

[pfft2001]

Question(s):
    1) Is it possible to change (or, ideally, eliminate) the 4 hour limit?  I
did not see it in the settings file(s) that the program manipulates.  Note
that Rick has access to the source code (for vrecorder); perhaps he could
check this for me.  I think the best solution would be if it were under user
control - i.e, you could set it to whatever you wanted or eliminate it
completely.  Also note, we *can* re-compile new versions of, e.g.,
vrecorder, and then use the "mount --bind" trick to "replace" the existing
(read-only) one on the flash media.  All this without having to do an "all
new" version of "the firmware".

    2) (Out of curiosity) Are there any other cmd line parameters to
vrecorder, other than the tag lookup into .vrecsettings?  I hope this can be
easily parsed out by looking at the source code, but of course, it'd be
really nice if there was actual proper documentation.

    3) A bug in the OSD built-in scheduler.  Background: When I "turn off"
an entry in the scheduler (generally, because I've moved it to my new
scheduler), what I do is to set the year to some year in the distant future
(like 2025).  This is to avoid deleting it, because setting these things up
from scratch is a pain and it seems like setting the year off in the future
is a good way to "put it on hold".  However, I've found over the last year
or two that the recordings mysteriously start running again.  I've now
figured out that it happens like exactly 1 year after you turn it off - and
then runs daily from there until you catch it and fix it.  It is as if the
scheduler only has a 1 year "look ahead" window - it simply can't fathom
anything farther away than that.

Anyway, it would be nice if it were possible to completely turn the
scheduler off.  Maybe all that's needed is to kill some process.  Could
someone knowledgable advise me on this?  Thanks.

    4) I noticed that the settings file contains this line:

   Video-Recorder-Advanced=0

    Anyone have any idea what this is about?  What would happen if I changed
    it?  And, related, are there any other settings I can put in this file,
    that are not currently present in the sysconfig.ini file?

Clarification on #1 above (post as first response):

The main reason why I don't like the 4 hour limit is that it is unnecessary
on modern filesystems.  It is a FAT filesystem based limitation that is
out-of-date now.  I do all my recording to cifs-mounted NTFS filesystems.

Further, the "turn around" takes time.  If you record, say, a 5 hour
program, you will lose about a minute of the show while the system goes
through the work of shutting down the first recording, finalizing the file,
starting up the next recording, and getting the next recording going.

Also, generally, because it is just one more thing that can go wrong;
specifically, because the process described in the previous paragraph (the
"turn around") is fragile and can cause the OSD to crash.  After seeing this
happen a few times, I have determined that it is some sort of memory issue -
that it is more likely to happen if there are extraneous processes (read:
extra shell sessions and non-core-OSD processes) running at the time.  When
I stopped running these extra processes, it pretty much (but not completely)
stopped crashing during the "turn around".

[heyrick]

Hey!

Sorry for the delay in getting back to you. Been busy and fuzzy-headed. My job is now daytime hours 10am-5.45pm (no more $%$(% $£*£&%ing NIGHTS!!!!!!!!!!  ) and it has been a bit of a shock coming home about the time I used to get up!

You have my email (don't you?). If you think I can help, nudge me! The latest episode of Kokoro Connect can wait...

1) Is it possible to change (or, ideally, eliminate) the 4 hour limit?  I did not see it in the settings file(s) that the program manipulates.  Note
that Rick has access to the source code (for vrecorder); perhaps he could check this for me.

I don't recall seeing it. To be honest, it looks as if vrecorder simply passes all the info to the codec.

The command line parsing is here:
  https://svn2.neurostechnology.com/linux-r3-main-app/trunk/vrecorder/src/ui/nvrecorderapp.cpp

And the interesting inside-guts-and-gore stuff is here:
  https://svn2.neurostechnology.com/linux-r3-main-app/trunk/vrecorder/src/ui/nvrecorder.cpp
  [especially function "NVRecorder::handleServerSelfStop", it looks like there are some hardcoded limits]


My pet hates that (probably!?) can't be fixed:

• Given we are writing AAC audio and H.263 frames; an option to use AVI (for this is possibly close enough to pass as DivX-like) would have been nice. I'm okay with the computers/phones as they'll play back .mp4 without problem, but a lot of older hardware based things only recognise AVI.
• My recordings are anamorphic. This means it is a 4:3 ratio picture containing a 16:9 frame. This is not unusual; older (non HD) satellite recorders output this when you use the "16:9Full" option (as opposed to "16:9Letterboxed"). Widescreen DVDs are this (and accordingly can be switched to "full" mode). All it needs is for the playback device to resize the video to the correct dimensions on playback. But for this to work automatically, you need the video file to identify itself as 16:9 accordingly. [well, it would at least fiddle the PAR in the mp4 header to reflect 16:9 attributes (actual values depend upon frame size)]. I wonder if I could hack vrecorder to do this at the end of a record? It'd be less hassle than pressing 'A' half a dozen times in SMPlayer for everything I want to watch!

I think the best solution would be if it were under user control

I wish there was a telnet-friendly version. How cool would it be if you forgot to record Meh so you excuse yourself to go to the toilet, fire up your Android phone, telnet to your OSD, and say "okay, record at this quality for the next 59 minutes".

vrecorder, and then use the "mount --bind" trick to "replace" the existing (read-only) one on the flash media.

Or for those of us with OSDng...

2) (Out of curiosity) Are there any other cmd line parameters to vrecorder, other than the tag lookup into .vrecsettings?

Looks like "temp" and "scheduler" are the options. [first link above, first function]

I hope this can be easily parsed out by looking at the source code, but of course, it'd be really nice if there was actual proper documentation.

A Real Nerd[tm] would say "the source is the documentation".

It is as if the scheduler only has a 1 year "look ahead" window - it simply can't fathom anything farther away than that.

Does the scheduler really even use the year? This might explain the symptoms you are experiencing? I don't know about you, but I find this code to be a bit of a mess. Not in how it is written, but more in that parts of it are scattered around the place and it is something of a game of "hunt the function" to follow it.
  https://svn2.neurostechnology.com/linux-r3-main-app/trunk/scheduler/src/scheduleritem.cpp

I tell ya, if I was unemployed (and thus had a lot of time and no money to do anything), I'd consider rewriting the UI from the ground up. Of course, I'd probably make just as big a mess, only it'd be my mess, ho ho. One thing I'd get rid of for a start is the MySQL stuff for the scheduler. A formatted data file would work - we're only recording (fairly small numbers of) user-specified recordings, this can be handled in a bunch of parse/sched functions. Anything else is just overengineered.
I would also look at exactly how much of the windowing system is necessary. That Arizona overruns the built-in 16Mb Flash is - frankly - obscene.
Take a look at https://www.riscosopen.org/content/downloads/other-zipfiles and you'll see the ROM image for the Beagleboard is around 2Mb, and the RaspberryPi build is also 2Mb. These are compressed, it's 4Mb raw. Add to that around 10Mb of resources (a lot of which could probably be dropped as much of that is example software). You would easily get an entire full installation of RISC OS into 16Mb. So why does a stripped-down Linux aimed specifically for putting into the OSD overflow? Is all that crap really necessary?

Hey - Neuros - the Beagle has a digital camera input. Don't suppose a tvp5150 would work here, eh?

Video-Recorder-Advanced=0
Anyone have any idea what this is about?  What would happen if I changed it?

I may be wrong, but I think this flags when you use the advanced recording setup, instead of the "Record for TV", "Record for PSP" sort of stuff; to indicate to the recorder app that it should look at all the supplied parameters instead of making assumptions.

Take a peek at:
  https://svn2.neurostechnology.com/linux-r3-main-app/trunk/neuros-qt4-libs/neux/settings/nvrecorddata.h

Well.....maybe it can do AVIs after all? Let me know if you decide to try that! I also think it would be kinda nifty to expose the quality option (economy/fine/superfine) as it doesn't seem as if the default settings control does that (I guess it picks depending on chosen bitrate or something?).

The main reason why I don't like the 4 hour limit is that it is unnecessary on modern filesystems.  It is a FAT filesystem based limitation that is out-of-date now.  I do all my recording to cifs-mounted NTFS filesystems.

Um... Some thoughts:

1. The FAT limitation is, IIRC, 4Gb minus a byte. This is not the same as four hours (depends on your bitrate!). Does it cut out at four hours exactly? With other quality options? Maybe your table of iframe pointers (that's what the finalise-TOC thing is, isn't it?) fills up. A quick look at a video file, it looks like it writes an iframe roughly every 15 frames. For me, there's 25fps so... 60 (secs) x 60 (mins) = 3600 secs/hour. x25 = 90000 frames/hour. /15 = approx 6000 iframes/hour. Maybe there's a limit to this?
2. You might record to NTFS. Does everybody else? That said, I need to know if you meant 4Gb or 4 hours.

Further, the "turn around" takes time.

Indeed. That's why long things I want to record are usually done manually. The Eurovision Song Contest, for instance, is usually recorded at a nicer bitrate (say 1500/2000kbit) for the songs. That's about the first half. When it comes to the voting, interval rubbish, etc, then I can knock the quality back to 1200kbit for the remainder of the programme. Not ideal, all in one file would be best, but it is a workable compromise as I can choose to abort during some of the blather stuck in for crappy new-Europe countries that can't handle a ~3hour live event without sticking adverts in...

When I stopped running these extra processes, it pretty much (but not completely) stopped crashing during the "turn around".

Indeed. The OSD is a little fragile in memory stakes. I note that the allocation to Linux is 34Mb. Do the codecs/DSP/etc really need to claim 30Mb of workspace? It is such a shame that part is closed, nobody can look to seeing how much space is given vs how much space is used.


Well, it's been... good grief, a month, since your post. How have you got on?


Best wishes,

Rick.

[pfft2001]

Thanks for posting.  There's no real hurry associated with any of this...

It'll take some time for me to digest your post completely, but at this point, I just want to say that it *is* 4 hours, not 4Gb.  It is set that way, based on an assumption of recording at full quality (2500 fps), which works out to 1Gb/hour.  But I always record at 1500 fps, so when I record for 4 hours, the file is just under 3Gb in size (literally, about 2950 Mb)

• .  And if I try to record for more than 4 hours, the first file cuts at 4 hours (which is, as noted, significantly smaller than 4Gb size) and it starts up a new file.
• Note: Throughout, I am using the "common man" definition (which I guess now is actually ISO standard) - where Mb means 1,000,000 and Gb means 1,000,000,000 - as opposed to the actually correct numbers which are exact powers of 2.  Just FYI...

P.S.  You could very well be right that this parameter is outside of our control - i.e., maintained and set in the non-open parts of the code.  You are correct that "vrecorder" is largely just a front-end to the underlying code that does the actual work.  I did some detective work and discovered that the actual work is done by a program called 'nmsd'.  I'm assuming that program is non-open.

[heyrick]

it *is* 4 hours, not 4Gb.

In that case, I'd be very inclined to point my finger at some sort of fixed limitation within the codec; as you proved by the smaller file of the lower bitrate.

• Note: Throughout, I am using the "common man" definition (which I guess now is actually ISO standard) - where Mb means 1,000,000 and Gb means 1,000,000,000 - as opposed to the actually correct numbers which are exact powers of 2.

AAAAAAAARGH!

Are you a Mac user, or using one of those suck-up-to-the-standards-bodies file managers under Linux? The only reason for promoting this incessent lie so your terabyte harddisc can be labelled as such while having a capacity of a lesser 931GiB. It was accepted as an ISO thing because nerds did kinda steal "kilo"/"mega"/"giga" for their own uses, but to apply the ISO definition to storage media is just a scam.
Here's why.
[soapbox rant!]
In computer storage, there is no base 10 component. Think about it. It's an 8/16 bit bus, returning one or more bytes from the storage media arranged (usually) in sectors of 512 bytes. 512*8 = 4096 bits. There is no way on earth you're going to divide that decimally. And even so, you aren't reading ten bit values, you're reading 8 bit values. Even flash chip erase windows are 32/64/128 kilobytes-in-the-old-school-style. In short, as disc sizes grew, this was a way to offer less as more. The difference between 250Mb and 250MiB is not so much, but the difference between 1TB and 1TiB is a whopping 93GiB - until not so long ago, computers came with harddiscs of that sort of size as standard.
There is no sane and logical justification, except to say "marketing ploy".
[/soapbox]

Just FYI...

I'm trying to switch to KiB/MiB/GiB, but as an hacker of the '80s and used to both Windows and RISC OS (which use the base two definitions), I usually interpret file sizes as power-of-two quantities.

I did some detective work and discovered that the actual work is done by a program called 'nmsd'.  I'm assuming that program is non-open.

I'm assuming "nmsd" is short for "nms daemon" (spelled "ae" thanks to those damn Christians corrupting Greek mythology and making demons something malevolent). If we take it a stage further, we could figure out the role "nmsd" plays to devise an acronym such as "neuros media services daemon". That sounds like something that oughta be in the source pile.

<clicky><clicky>

Oh look... https://svn2.neurostechnology.com/linux-r3-main-app/trunk/nms/src/

[side note: I wonder if this is the place to fix that annoying "can't play this video file so will just go ahead and play something else instead" quirk?]


The huge main play loop in https://svn2.neurostechnology.com/linux-r3-main-app/trunk/nms/src/server/server-play-nms.c contains some amusing comments; most by a guy called Nero.

[heyrick]

Bingo!


You can hug me later. Or send me JPEGs of cute (as in HelloKitty, not as a euphemism for naked) Japanese girls. Your choice.


Anyway. The code you are looking for is:
/*
 * Recording finalization is using too much memory with low quality sometimes and the system crash
 * due to out of memory. So limit the maximum of recording length to hack it for now.
*/
#define MAX_LENGTH (4 * 60 * 60) // 4hours

It is in nms; and the source is at https://svn2.neurostechnology.com/linux-r3-main-app/trunk/nms/src/server/server-record-nms.c


I don't think anything can be done above the unpleasant video resize; namely if you are recording 640x480, it looks like the video (for PAL) goes:
  720x576 (input) -> 720x288 (inside) -> 640x480 (codec)
It would be better if we could work without losing half the vertical resolution; this is one of the reasons I record anamorphic and not letterboxed!

We might, however, be able to fiddle the appallingly low audio sampling rate (16000Hz it would seem). The AAC output is rather astonishingly good given this sample rate is only twice that of a basic telephony codec, and about a third of regular (41kHz) audio sampling.

Good grief, is the DM320 that underspecified?!?


Anyway. There's your four hour limit. Have fun trying stuff that doesn't cause it to crash!

[ChadV]

The huge main play loop in https://svn2.neurostechnology.com/linux-r3-main-app/trunk/nms/src/server/server-play-nms.c contains some amusing comments; most by a guy called Nero.

Nero?  I'm pretty sure that's nerochiaro, former Neuros employee.  I'm not sure if he still lurks or not.

[pfft2001]

A couple of things:
1) Regarding the 4 hour limit.  So, it has nothing to do with the FAT 4G file size limit?  That was just a coincidence.  It has to do with running out of memory (and the system crashing) when finalizing the file.  Note that because of the MP4 file structure, you have to pass the entire file through in order to finalize, so it makes sense that the larger the file, the more memory it will take to do it.

2) Regarding the Gb vs. GiB thing.  I'm not using any file manager.  GUI file managers are for wimps.  I  just use variations of the 'dir' command ('dir' under CMD.EXE or the equivalent 'ls' command under Unix flavors) and that displays the sizes of files in bytes.  If I see a 10 digit number that starts with 2950, that says "just under 3 Gb to me".

I understand, and empathize with your little rant, but it is also true that if you see a file size like:

3,098,123,456

it makes sense to be able to say that it (the file) is "just over" 3 Gb.  (Even though that statement is, technically, untrue...)

[heyrick]

1) Regarding the 4 hour limit.  So, it has nothing to do with the FAT 4G file size limit?  That was just a coincidence.

See? The Asians have a dislike for the number 4 for a reason. [see Tetraphobia (Wiki)]

It has to do with running out of memory (and the system crashing) when finalizing the file.  Note that because of the MP4 file structure, you have to pass the entire file through in order to finalize, so it makes sense that the larger the file, the more memory it will take to do it.

Which is what I was thinking with the I-frame index. Good God, how did such a shonky video format ever become a standard? Why can't the start of one chunk point to the previous and next? It'd only be 16 bytes for two 64 bit values (or could use 32 bit values if it's just an "offset to prev/next chunk" (instead of absolute position). This would also mean the files could be playable if incorrectly/not finalised.

I'm not using any file manager.  GUI file managers are for wimps.

Hahahaha! You're a dying breed, you know? I bet you love Unity about as much as I do.

3,098,123,456
it makes sense to be able to say that it (the file) is "just over" 3 Gb.  (Even though that statement is, technically, untrue...)

I handle that by saying "it's about 3gb", and I would work out the size in MiB/KiB/bytes if it came to actually mattering that much; 99% of the time the only comparisons I have are will this stuff fit on a DVD-R (<=4488MiB) or do I have enough space on this SD card for the next episode of Joshiraku?...

I only get annoyed when I look in the supermarket and see 500Gb harddiscs, and I know well that if I plugged it in, my computer would tell me something rather different; and worse, as I pointed out, it isn't as if harddiscs have any denary component, so using non-binary for the sizes is just a crappy thing to do...

Anyway. Hope you can rebuild (or maybe binary-hack) your nmsd to see how far you can push it. Let me know how you get on.

[ChadV]

3,098,123,456
it makes sense to be able to say that it (the file) is "just over" 3 Gb.  (Even though that statement is, technically, untrue...)

I handle that by saying "it's about 3gb", and I would work out the size in MiB/KiB/bytes if it came to actually mattering that much; 99% of the time the only comparisons I have are will this stuff fit on a DVD-R (<=4488MiB) or do I have enough space on this SD card for the next episode of Joshiraku?...

I have "ll" aliased to "ls -lh --color" on most of my systems.  No conversion necessary.