Help with 4k

category: offtopic [glöplog]
Code: static const char *sync_track_name(const char *base, const char *name) { static char temp[FILENAME_MAX]; strncpy(temp, base, sizeof(temp) - 1); temp[sizeof(temp) - 1] = '\0'; strncat(temp, "_", sizeof(temp) - 1); strncat(temp, name, sizeof(temp) - 1); strncat(temp, "_", sizeof(temp) - 1); strncat(temp, "data", sizeof(temp) - 1); return temp; } void sync_save_tracks(const struct sync_device *d) { FILE *file = fopen("sync.h","wt"); int len = 0; FILE *fileread = NULL; int count; int i; for (i = 0; i < (int)d->data.num_tracks; ++i) { const struct sync_track *t = d->data.tracks[i]; save_track(t, sync_track_path(d->base, t->name)); //read trackfile fprintf(file,"//data from rocket track: %s\n",sync_track_path(d->base, t->name)); fileread = fopen(sync_track_path(d->base, t->name),"rb"); fseek( fileread, 0, SEEK_END); len = ftell(fileread); fseek(fileread, 0, SEEK_SET); count = 0; //write data fprintf(file,"unsigned char %s[%d] = {\n\t",sync_track_name(d->base, t->name),len); while (!feof(fileread)) { unsigned char ch = fgetc(fileread); fprintf(file,"0x%02x,",ch); count++; if ((count & 15) == 0) fprintf(file,"\n\t"); } fprintf(file,"\n};\n// end of %s\n",sync_track_path(d->base, t->name)); fprintf(file,"\n"); fclose(fileread); } fclose(file); }

My current code. Dumps GNU Rocket track files to memory arrays, which I then load using a very thin memio wrapper based on the FILE* based stdio functions. Works brilliantly so far.
added on the 2014-02-25 03:11:27 by mudlord mudlord
I noticed that you changed the fix point values storage to floats.
I actually have originally implemented the tiny player to have different data array for each field of the event (time, position, type) - the idea is to have a better data compression rates.

Having the data in a arrays, I tried several things like storing deltas, fix-point, even tweaking values to have more 0 bits (ie 1024 instead of 1023). All those help but the major compression advantage came from using fix point for value storage, those get compressed much better compare to float arrays.
added on the 2014-02-25 07:19:58 by TLM TLM
My code is intended for raw unmodified GNU Rocket, I didn't do any mods to use fixedpoint, etc.
added on the 2014-02-25 07:38:50 by mudlord mudlord
IIRC, spookysys just added some editor-support to truncate mantissa bits in the editor to save space in Texas. I'm sure something like that could be done upstream as well.
added on the 2014-02-25 09:25:11 by kusma kusma
I did try to crack out some code. It's not super-useful in the current state, but it some kind of sketch at least.
added on the 2014-02-25 11:12:59 by kusma kusma
While post-release squeezing Dodecaplicit below 10k (just for future stuff - cut ~1kb from the rocket data alone compared to the already binary representation used back then at release), I took another approach than TLM, which is more flexible and convenient (essentially the same API as the standard one), maybe compressing a bit better than what i can see in the code snippet, but the flexibility is obviously also costing some bytes. And it's not so much a code generator as TLMs.
So my version is meant for 8-64k, while TLMs is the 4k suited version (I would probably still go completely custom for 4k, but that's another story)

I'm storing per element type (keys, types, values), delta encoding keys (actually types are packed into the 16bit keys after that), while values are quantized floats. Quantization is something you specify per variable at creation time, and not in neither editor nor player code (basicly just the import from the editor).
Variable names (and quantization) are #defined away in player version, and instead only the order of initialization is used.

So I got a compact format that compresses well, smaller player code and except the optional quantization bits and the data pointer the same user code as the "demo" version.

Not too happy about the current state of the source though - it has become a bit too much #define hell, should probably move the player to it's own file..
added on the 2014-02-25 14:03:39 by Psycho Psycho
Psycho: Cool, I'd love to see the code.

Another reasonably small approach is how spooky did the replayer in Texas: at init-time, expand all sync-data to piecewise linear segments sampled at a fixed frequency. He used 50hz, so that's some pretty huge buffers, but I think by doing piece-wise quadrics instead, it only really needs one sample pr row. That gives you a simple lookup for a simple evaluate function, give random row-access without worries. But I do suspect that it costs a few bytes more than interpolating once per frame.

And besides, I must admit: even when size isn't an issue, I very rarely do any thing else than reading at the current row.
added on the 2014-02-25 18:08:45 by kusma kusma
I've found some bugs in kusma's usync implementation, here's a version that exports correct values (floats instead of decimals) and finds the right row during playback:

Code:#include "usync.h" #include <math.h> #ifdef SYNC_PLAYER static int usync_rows[SYNC_TRACK_COUNT]; float usync_values[SYNC_TRACK_COUNT]; void usync_update(float t) { int i, row = (int)floor(t); for (i = 0; i < SYNC_TRACK_COUNT; ++i) { int pos; float mag, x, a, b, c, d; /* empty tracks should not be neccesary! */ if (!sync_data_count[i]) { usync_values[i] = 0.0f; continue; } /* step forward until we're at the right key-frame */ while (usync_rows[i] < (sync_data_count[i] - 1) && row >= sync_data_rows[sync_data_offset[i] + usync_rows[i] + 1]) { usync_rows[i]++; } pos = usync_rows[i] + sync_data_offset[i]; /* we need a segment to interpolate over */ if (usync_rows[i] == sync_data_count[i] - 1) { usync_values[i] = sync_data_values[pos]; continue; } /* prepare coefficients for interpolation */ a = sync_data_values[pos]; mag = sync_data_values[pos + 1] - sync_data_values[pos]; switch (sync_data_type[pos]) { case 0: b = c = d = 0.0f; break; case 1: b = mag; c = d = 0.0f; break; case 2: b = 0.0f; c = 3 * mag; d = -2 * mag; break; case 3: b = d = 0.0f; c = mag; break; } /* evaluate function */ x = (t - sync_data_rows[pos]) / (sync_data_rows[pos + 1] - sync_data_rows[pos]); usync_values[i] = a + (b + (c + d * x) * x) * x; } } #else /* !defined(SYNC_PLAYER) */ #include <stdio.h> #include "modified-rocket/sync.h" #include "modified-rocket/device.h" struct sync_device *usync_dev; float usync_time = 0; void usync_update(float t) { usync_time = t; sync_update(usync_dev, (int)floor(t), &usync_cb, usync_data); } int usync_init(void) { usync_dev = sync_create_device("sync"); return sync_connect(usync_dev, "localhost", SYNC_DEFAULT_PORT); } void usync_export(void) { int i, j; int offset = 0; FILE *fp = fopen("E:\\blu-flame.org\\nordlicht2014-intro\\sync-data.h", "w"); if (!fp) return; /* header-guard */ fputs("#ifndef SYNC_DATA_H\n#define SYNC_DATA_H\n\n", fp); fputs("enum sync_tracks {\n", fp); for (i = 0; i < usync_dev->data.num_tracks; ++i) { struct sync_track *t = usync_dev->data.tracks[i]; fprintf(fp, "\tSYNC_TRACK_%s = %d,\n", t->name, i); } fprintf(fp, "\tSYNC_TRACK_COUNT = %d\n", usync_dev->data.num_tracks); fputs("};\n\n", fp); fputs("static const int sync_data_offset[SYNC_TRACK_COUNT] = {\n", fp); for (i = 0; i < usync_dev->data.num_tracks; ++i) { struct sync_track *t = usync_dev->data.tracks[i]; fprintf(fp, "\t%d, /* track: %s */\n", offset, t->name); offset += t->num_keys; } fputs("};\n\n", fp); fputs("static const int sync_data_count[SYNC_TRACK_COUNT] = {\n", fp); for (i = 0; i < usync_dev->data.num_tracks; ++i) { struct sync_track *t = usync_dev->data.tracks[i]; fprintf(fp, "\t%d, /* track: %s */\n", t->num_keys, t->name); } fputs("};\n\n", fp); fputs("static const int sync_data_rows[] = {\n", fp); for (i = 0; i < usync_dev->data.num_tracks; ++i) { struct sync_track *t = usync_dev->data.tracks[i]; fprintf(fp, "\t/* track: %s */\n", t->name); for (j = 0; j < t->num_keys; ++j) fprintf(fp, "\t%d,\n", t->keys[j].row); } fputs("};\n\n", fp); fputs("static const float sync_data_values[] = {\n", fp); for (i = 0; i < usync_dev->data.num_tracks; ++i) { struct sync_track *t = usync_dev->data.tracks[i]; fprintf(fp, "\t/* track: %s */\n", t->name); for (j = 0; j < t->num_keys; ++j) fprintf(fp, "\t%.6f,\n", t->keys[j].value); } fputs("};\n\n", fp); fputs("static const unsigned char sync_data_type[] = {\n", fp); for (i = 0; i < usync_dev->data.num_tracks; ++i) { struct sync_track *t = usync_dev->data.tracks[i]; fprintf(fp, "\t/* track: %s */\n", t->name); for (j = 0; j < t->num_keys; ++j) fprintf(fp, "\t%d,\n", t->keys[j].type); } fputs("};\n\n", fp); fputs("#endif /* !defined(SYNC_DATA_H) */\n", fp); fclose(fp); } #endif

(I haven't used github before, so I don't really know how to contribute/patch. Please forgive this)
added on the 2014-07-21 22:06:05 by xTr1m xTr1m
Of course, the hardcoded paths should be replaced accordingly... ;)
added on the 2014-07-21 22:07:09 by xTr1m xTr1m
Thanks! I wasn't aware anyone tried to use this version yet, so it's cool to see patches :)
added on the 2014-07-22 00:13:18 by kusma kusma