WAV files and Synchronization Problems

[jjthrash]

Howdy all,



I'm having problems with synchronization of WAV audio with animation.



I have a very long animation, along with a WAV file which is supposed to be synchronized with the animation. I've got it working by doing an FDTDefineSoundWAV in the 0th frame, and by doing the normal animation frame by frame.



However, on different computers the animation plays back at different speeds, and therefore is not necessarily synchronized with the sound. Is this normal? I've set it to the default 12 frames per second. It seems that seconds are longer on some machines than on others. :)

[johnie]

Unfourtanatly this is normal for the Flash Player, playing at different speeds on different machines depending on the resources of the machine.

Use a MP3 and synch to that and the player will throw out the lagging Frames.

[jjthrash]

Apparently I can also do that with ADPCM data, which is currently nicer for our environment. Is there any sample code for taking ADPCM data and figuring out what data to put in which frame?

If there are 11025 samples per second, and the movie plays at 12 frames per second, does that mean I should put 11025/12=918.75 samples per frame? This seems difficult, you know, dividing frames into fractions.

Any ideas?

Thanks,

Jimmy

[JAEzell]

The SWFSource SDK has an example with MP3 data streaming:

http://www15.brinkster.com/JAEzell

the FAQ section of the site also has info on converting MP3s. If you have any other questions, post again.

--Jesse

[JAEzell]

PS: Include full frames only in stream blocks and use delay-seek values to instruct the player as to the delay between blocks in case of a jump command being issued.

--Jesse

[jjthrash]

Thanks for the info. It's been very helpful.

However, I really need to use ADPCM instead of MP3, because compression time is too high with MP3. How can I use your fine FlashMP3Encoder and translate it to something I can use with ADPCM?

The MM SDK includes ADPCM stuff, and it seems that I can get the data, but I don't know exactly what to do with the data in relation to the frame of the animation. Do I just group them together according to what frame they go with?

So, if my movie is 12 fps, and my sound is 11025Hz, would I do the following (in pseudocode):

frame[i] <- SoundStreamBlock( all data such that the sample's occurance is >= i*(1/12) && < (i+1)*(1/12) );

?

Once again, thanks for the help, and I apologize for my obvious lack of experience with sound data. :)

[JAEzell]

The number of samples per frame really doesn't matter (as long as the header is correct), but it is best to include the least amount possible to maintain the stream info that would be used assuming the SWF plays at the exact speed it loads. Your pseudo code looks fine.

--Jesse

[lifemachine]

hi jjthrash

It seems that we are running into the same problems.
I 'm using ADPCM && the MM SDK to create the blocks and I actually managed to create a good streaming ... NOISE ... I must be doing something wrong in the header.
If you can give me a tip or share your code it will be highly appreciated.

[jjthrash]

I haven't actually been able to get anything yet. I have a WAV file I'm trying to convert. I used the code from FDTDefineSoundWAV to get the ADPCM data, but it seems I'll have to do weird unnatural things in order to get it to work right with the SoundStreamBlocks.

So, as a test, I put all the data in one SoundStreamBlock, at frame 0, and it still doesn't work. The Flash Player steals all possible CPU cycles and reduces the response of the computer to a minimum, besides not playing the sound.

Shouldn't putting all the data in one block work? I know it's bad for streaming, but at this point I'm just trying to get it to work.

Thanks,

Jimmy

[lifemachine]

I actually got it to play something, unfortunately it was noise, I also did the same "one block thing", with it, the playback seamed to have some connection to the original audio, only it played slower ???... weird ****. I suspect that the streaming part of the SDK is not working, specially taking into consideration that the streaming sound doesn't work in generator as well.

[JAEzell]

Yes. Macromedia's Streaming code is incomplete.

--Jesse

[jjthrash]

I found the bug in the MacroMedia SDK.

In FDTSoundStreamHead2::WriteToSWFStream, there are the following lines:


FSWFStream body;

body.WriteByte((U32)mixFormat);
body.WriteBits((U32)compression, 4);
body.WriteBits(0, 2);
body.WriteBits((U32)size, 1);
body.WriteBits((U32)soundType, 1);
body.WriteWord((U32)count);

_SWFStream->AppendTag( stagSoundStreamHead2, body.Size(), &body );

The line that reads body.WriteBits(0, 2) should put, instead of 0 (which means 5.5kHz), whatever format your data is in, in my case 44kHz, or 3.

Jesse, I think I like your SDK better by virtue of the fact that it makes sense and is not a hacked up piece of ad-hockery.

[JAEzell]

Yah...I would never have wrote it if the Macromedia SDK worked :-).

--Jesse

[lifemachine]

You did it.

It works beautifully with one block. I still have a problem though when I break it into more than one frame. The first block always plays fine, from there on ... well ... hhmmm ... I get noise.
Did you manage to make it work?

I did a straight forward implementation according to the documentation, but...

Here is what I do.

I set the header with the avarage samples per block/frame:

int sampleCount=sound->NumSamples()/frameNum;
FDTSoundStreamHead* head = new FDTSoundStreamHead( sound->Format(), sound->SoundType(),sampleCount);//sound->NumSamples()

and than I loop to create a block per frame:

std::vector<FDTSoundStreamBlock*> soundStreamBlocks;
int blockSize=pcmData.size()/framesNum;
int curElm=0;
int i=0;
FDTSoundStreamBlock* tmpBlock;
// loop for all frames until the last one
for (i=1; i<framesNum; curElm=(i++)*blockSize){
tmpBlock=new FDTSoundStreamBlock(blockSize,&pcmData[curElm]);
soundStreamBlocks->push_back(tmpBlock);
}

// do last frame block
blockSize=(pcmElmsNum-curElm);
soundStreamBlocks->push_back(new FDTSoundStreamBlock(blockSize,&pcmData[curElm]));

What do you think?

[Jestryn]

FYI I had the same problem a few months back. My WAV samples generated only noise in the animation. I eventually figured out that it was due to my using 8 bit samples. Indeed, FDTDefineSoundADPCM doesn't support 8 bit, despite the doc. Automatically converting to 16 using the following simple conversion fixed my problem:

for (i = 0; i < sampleCount; j++)
{
value = (short)p8BitSamples[j];
value = value - 128; // convert to a signed quantity
p16BitSamples [j] = value << 8;
}

Also, as HFSound uses FDTDefineSoundADPCM, the high-level code cannot be used to add sound without compression. FDTDefineSoundADPCM crashes if you pass it FDTDefineSound:: NO_COMPRESSION. So I assume that the only way to use compression-less sound is by using FDTDefineSoundWAV, which requires a pointer to a FILE. If you're using stream, you will have to compress your sample.

And as you were discussing, if you look at FDTSounds.cpp, in the
FDTSoundStreamHead::WriteToSWFStream or
FDTSoundStreamHead2::WriteToSWFStream routines, they both say:
body.WriteBits(0, 2); // sample rate 0=5.5, 1=11, 2=22, 3=44
Note that the 0 is hardcoded. I personally ignored this issue; despite the tagSoundStreamHead being always wrong when I parsed the resulting file, the sound worked just fine regardless.

Also, make sure you're not deleting the sound object before it gets written to the movie. If it is, then it'll end up as noise.

SDK used: format 4, dated 3-21-00

Hope this helps!

[jjthrash]

Sorry I can't help with the ADPCM problem you're having. I've given up on it and moved to MP3.

So, I have a new problem. The data gets entered into the SWF file properly. That is, 1:06 worth of MP3 sound gets split up into 797 frames, which is correct. My drawing data starts at 27 frames, which is correct.

However, even though the sound plays properly, the drawing doesn't start until long after it's supposed to, and doesn't play at nearly the proper speed. Is there a reason for this, or is this just another Flash annoyance?

[jjthrash]

More clues:

My MP3 file is 1:06 long. My drawing starts 30 seconds after it should start (at 0:30) and goes for 1:06 seconds, lasting until 30 seconds after the sound data ends (1:36). So, it looks like the sound data is both stretched and translated, time-wise, since the drawing portion should only last around 10-15 seconds.

[JAEzell]

Could you post the sample file somewhere.

--Jesse

[jjthrash]

Certainly:
http://theguys.dyn.dhs.org/file.swf

[JAEzell]

Check your avg. samples per frame caculation (the value stored in the stream header). If the value is incorrect, it changes the speed of the SWF to match for some reason. Another undocumented feature of the SWF File Format.

The calculation should be:

const double mspf_mp3 = 26;
double avg_samples = (double)((1000.0/frameRate_in_fps)/mspf_mp3)*(double)(pcmFrameSize);

(adapted from SWFSource SDK code)

--Jesse