[MagicLantern] raw2cdng – Cinema DNG

i thought, cinema dng could be the right format to open the rawdata into the bigger grading world, for example speedgrade and resolve. now we have to see. for now there is this tool for windows and the cdng-converter rawmagic for mac written by thomas worth. my github link was only for first discussion purposes. you will find the CinemaDNG-Dialog in the Magiclantern-Forum.
Resolve_16Bit_magiclantern_cinemadng

important!

– its just a raw to cinema dng converter – its NOT debayering!
– needs .net 4.0 Runtime – here
– made for windows-systems – not tested under emulators/virtualizers.
– its coded from scratch and has nearly nothing to do with raw2dng!
– this is a beta! need your response!


new Version 1.5.0 is in a new blog entry

blog.phreekz.de raw2cdng 1.5.0 magiclantern


raw2cdng10


version log and informations

11.06.2013 – v1.0 – fixes so far
– batchmode integrated
– blacklevel, whitelevel, colormatrix read from rawdata
– assuming body by first colormatrix-value
– 12bit linear built in (lsb 2bit cropped)
– 2gb crash fixed
– lowering whitelevel by 15% – maybe a helper for pink highlights
– tried to fix the slash/backslashproblem for pathes

11.06.2013 – v1.1.2 – fastfix
– if blacklevel wasn’t found in raw – it turned zero – fixed that to 2037
– blacklevel for 12bit was too high (/2 -> /4)
– stripByteCounts calculated now from resolution and bitsperPixel – reading the raw from newer ml-builds

14.06.2013 – v1.1.6 – possibly EDL/XML fix
– inserted Reelname-Tag with 8 Chars
– changed filename of pictursequence to section filename and sequencenumber with a dot.
– allowed for filename are only a-z and 0-9, other symbols will be deleted

20.06.2013 – v1.1.8 – 2nd attempt – EDL/XML fix
– put framerate and timecode into cdng-header
– reelname fix – nothing happend if filename is shorter than 8 chars

RESOLVE ROUNDTRIP IS WORKIN !
23.06.2013 (!) 1.1.8 was not the solution for functional roundtrip BUT peoplemerge found, how to solve the problem – you have to fill up the ReelNo-Attribute in Resolve to get unique Filenames for xml/edl-roundtrippin‘ – please look into his video at ~7:10

27.06.2013 – v1.1.9 – 10bit log
– 10bit log (!) experimental – first trials (red tint, but adjustable with red gain to ~0.86)
– pathnaming -> timedate of creation -> [yymmdd_HHmm]_[REELNAME]
– filenaming -> [REELNAME].[sequenceNo].dng
– shortened filenames (M12-1305.raw) randomly filled up to 8 symbols f.e. M121305T
– fixed GUI-freeze – worker-threads

(!) 10bit log only works for bodies with blacklevel higher 2000 (!)
next patch will have a lut for blacklevels around 1024 (!)

Advantages of less bitdepth (on 1280*544px)
– 16bit (filled up 14bit) – 1,37MB per frame
– 12bit (2bit shifting) – 1,02MB per frame (-26%)
– 10bit/log (lut) – 0,86MB per frame (-38%)

30.06.2013 – v1.2.0 – splitted Files
– splitted Files support (you only have to drag the raw-file)
– splitted Files – please reply if it’s working flawless.. 🙂
– import-bug fixed – if path/filename was shorter than 23chars it didnt imported file

01.07.2013 – v1.2.1 – selectable static outputPath
– you can select a outputfolder
– means, you can transcode straight from the card
– the settingsFile is json-serialized saved in user/userX/appdata/roaming
– randomly added chars are now calculated from the existent name -> unique name

13.07.2013 – resolve 9.1.5 supports 14bit-dng
– i’m kind of pissed. instead of optimizing their debayer-modul (pink fringes all over the world), seriously reading/using the Reelname for xml, doin‘ some badpixel-support, noisereduction-tool or manual devignetting-sliders they have done the simplest thing. ok, its a free software – and i hope, they build this all into the newer v10-version – but thomas worth, a1ex, g3gg0, iliasG, fatpig and lots of other coders did a way better (free) work to make a handy environment to compete the ML_RAW-revolution. I#m now a little bit tired thinking about optimizations – by now i’m doin a pause – rethinking this tool – there are three things: simplifying convert-process, lowering datasize and opening as a commandline-tool. Ahh, forgot one challenge: accurate colors/whitebalance/colormatrices/LUT – we re talking about it in the ml-forum – in Speedgrade the problem is obvious.

11.08.2013 – v1.3.0 – whitebalance from cr2
– as by now i wont maintain raw2cdng. all things happening from now on are tests of functionality for the forthcoming mlv2cdng. all opinions and experiences are welcome, please feel free to write them here into the blog or into the magiclantern-thread
– GUI slightly changed
– this version’s got CR2-whitebalance-support. do a photo right before the movie record and put this one into the same path as the raw-file. name it as the rawfile. raw2cdng will recognise it automatically and read the RGGBValues asMeasured (from makernotes) and turn it into the whitebalance-data for the dng-files. ITS ONLY TESTED WITH 5DIII CR2-Files!

Example:
M02-1810.RAW
M02-1810.CR2

If you recorded a whole set of same whitebalanced stuff, you name this file ALL.CR2 and all RAWs will get this whitebalance-value – except files havint their own CR2.

Example:
M02-1631.RAW (WB from ALL.CR2)
M02-1635.RAW (WB from ALL.CR2)
M02-1810.RAW (WB from the according CR2)
M02-1810.CR2
M02-2012.RAW (WB from ALL.CR2)
ALL.CR2

09.11.2013 – v1.4.0 beta – experimental
This one’s is a first usability-test for photoshop, speedgradeCC and resolve10 users. by 90% stripping wont work. multithreaded. lowering blacklevel by 90%-button. changeable UniqueModelID for ACR-Profiles. framerate fixed. .look-file inside for speedgradeCC.

24.11.2013 – v1.4.5 – multithreading and mlv
* fixed modell-tag
* multithreading working with stripped files (tested with 1 strip)
* whitelevel back to static 15.000 (16bit)
* speedgrade-look – changed multiplicator to 16
* mlv support (not native, but via mlv_dump)

put mlv_dump.exe into the same path as raw2cdng. it will be recognised on start and shown in the title. now you can drag&drop mlv-files. they will be rewritten to raw after dropping. raw-files will be saved in the same path as mlv_dump.exe

24.11.2013 – v1.4.6 – empty modelID-Crash fixed
* crashed, when modelID-Field was checked but empty
* switchable debug-Log (saved in according .log-File in same path)

raw2cdng description

Since 1.4.8:
For ACR-Applications (AE,Photoshop), Speedgrade and Resolve please use
*(5) 16bit linear
*(6) no partymode, Gamma 1.0
*(4) enable and fill in the according Model ID for right profiles. Example: Canon EOS 5D Mark III
*(+) for Speedgrade you have to use a calibration-lut, there’s an example in v1.4.8

For Premiere CC (works with Resolve as well):
*(5) 12bit linear
*(6) partymode enabled, Gamma 1.0
*(4) Model ID not needed.

raw2cdng.1.4.8
04.12.2013 – v1.4.8 – partyMode and gamma
* to be honest, lets make party, because its the first step using cdng files straight inside premiere CC7.1
* set/change gamma to 1.0 – its built in for testing purposes.
* filename-prefixes are back on dng-files
* possibly stripped-files-bug fixed (didnt checked it with more than 3 strips)
* adjusted .look for speedgrade to new filedata
* inside premiere give the cineon-converter a try

09.02.2014 – v1.4.9 – audio from mlv
* audio recreation from mlv-files
* copied and renamed into Output-Footage-path
* in premiere just import the dng-sequence, the audiofile will be recognised automatically
* newest mlv_dump.exe from g3gg0 is inside the zip

19.02.2014 – v1.4.9.3 – mlv_dump fix
* wrong characters in path or filename leads to failure. fixed.
* modeltag and uniquemodel are separated (@Andy600 told about problems in cinelog)
* optimizing some things (fi tiff-dng-tag-creation)

12.03.2014 – v1.4.9.5 – native mlv with audio
* warning. test-version. tell all problems, bugs

16.03.2014 – v1.4.9.6 – bug fixing
* timecode set to 00:00:00:00
* framerate in dng fixed
* mlv-batchmode fixed
* framesCount in GUI fixed

21.03.2014 – v1.4.9.7 – pink highlights
* this is a testversion, if pinkhighlight-fix works with any body
* logarithmic approaches freezed
* partymode is now on by default, no click.
* raw-bug fixed (fps.denominator was the bad boy)


raw2cdng

download

v0.93 – not functional preversion for discussion
02.06.2013 – v0.99 – initial working version
11.06.2013 – v1.0 – see fixeslist above
11.06.2013 – v1.1.1 – see fixeslist above
11.06.2013 – v1.1.2 – see above
14.06.2013 – v1.1.6 – attempt to fix xml/edl
20.06.2013 – v1.1.8 – 2nd attempt to fix xml/edl
27.06.2013 – v1.1.9 – see above
30.06.2013 – v1.2.0 – splitted Files
01.07.2013 – v1.2.1 – output Folder
11.08.2013 – v1.3.0 – CR2-whitebalance
09.11.2013 – v1.4.0 beta – experimental multithreaded
24.11.2013 – v1.4.5 – mlv – speedgrade-look inside
24.11.2013 – v1.4.6 – empty modelID-Fix
04.12.2013 – v1.4.8 – partymode
09.02.2014 – v1.4.9 – audio
19.02.2014 – v1.4.9.3 – mlv_dump fixed
12.03.2014 – v1.4.9.5 – native mlv-support
16.03.2014 – v1.4.9.6 – bugfixing mlv-code
21.03.2014 – v1.4.9.7 – pink highlights

version 1.5.0 is on new blogentry!











Dont forget to donate the Magiclantern team (and maybe then me :)) This button is for donations for my job. But, ML did a fabulous job on enabling this hell of a gem! regards chmee


ToDo

– vertical lines banding (solved in raw2dng, but have to implement it)
– compression (important or not?)

done!

audio from mlv
threading in conversion – speeding up the thing.
static outputfolder
splitted files (raw r00 r01 etc pp) support
fix 2gb crash
embedding 12bit linear as well
embedding other color matrizes than 5DIII
– pink highlights (problem longer known, dcraw user saying, lowering the saturationlevel is the key. satlevel=whitelevel? with ETTR enabled in Magiglantern it shouldnt be no longer a problem)
there was a change in raw.c code – the stripbytecounts changed – have to fix that. (urgent!)
whitebalace refining via CR2-makernotes v1.3.0

ACR_CinemaDNG_magiclantern_raw

62 Kommentare

  1. This is my favorite app so far. Very excited for v1.0 for batch processing and larger files.

  2. Sanx for the program!

    >embedding other color matrizes than 5DIII
    If you want, i can do 6D raw shots, tell what exactly should i do.

    >- crashes on files >2gb (didnt thought about it)
    maybe it try allocating to much RAM at once?

  3. Thank you very much! After learning this RAW workflow much better i will invest in a decent camera like the 5D MKIII.

    It’s very much appreciated this work!

    To be able to use Cinema DNG is great!

  4. Hi, I was wondering if it would be possible to add 8 bit conversion. I thought it would be beneficial for those who want to do a quick turnaround using adobe’s dng importer for Premeire. It only reads 8bit dngs.
    Thanks!

  5. thank you for the idea – but.. as the dng-importer for premiere is officially not supported, it would be a drawback to go down to 8bit losing all benefits of dng. this can be done with all other converters..

  6. Hey hey, thanks for your amazing tool! But I’m afraid, I’m not able to export a XML from Resolve, which I’d like to use for Resolve->Premiere->Resolve Roundtrip.
    I get the message „The clip „xyz“ cannot be exported because it’s a frame based clip“. Also the interpreted FPS is wrong. It’s default on 24, but everything’s shot on 25.
    I gave RAWmagic a try on my MacBook, everything’s fine. FPS is on 25, XML export works like a charm. I’m using Da Vinci Resolve Lite 9.1.5.
    Any idea? I’d really like to use raw2cdng on my Windows Workstation 🙂

  7. thanks for your words.

    @G.Guggi – dont use the simple xml-export but the „Deliver“-Panel with the „Final Cut XML Roundtrip“ Preset.

  8. @chmee ok thanks for your hint, will try that. just out of curiosity: are you planning to change things, so the normal export XML process will work? my guess it’s only metadata that are different?

  9. as you read the error message – it cant export frame-sequences – i’m surprised, on mac this works.. there is some talking (in the ml-forum) about the naming of path and/or dng-files – if theres a C in front of the numbersequence it will be „some how“ different imported by resolve. Actually my coding-netbook crashed with all files (including deleting all files) – luckily all is recovered – but dont expect another update in the next days.

  10. Great app!!! Thank you so much!

    Which format is best for Davinci Resolve 12bit linear or 16bit linear?

    Any suggestions for good LUTs. I tried different ones but they are made for ohter cameras and i see little color shifts.

  11. In Davinci Resolve i got
    with 10b log a magenta tint.
    With 12b or 16b linear the clips are very dark.
    Also the LUTs from Arri, Hudson etc. are for Log to 709 conversion.
    Where can i get LUTs for Linear conversion?
    Or what am i doing wrong?

    In the Resolve Clip Raw Settings i have BM selected.

  12. thx.

    @sam
    both are working great – and the 12bits disandvantage (cutting lower bits) are practically in normal editing no problem. please decide yourself with two different bitdepths.

    @both – colorshift
    please try the v1.3 with the cr2-picture made right before the movie-recording – it neutralizes colorshifts barely completely – the 10bit-mode is only working with bodies with a blacklevel ~2048 – see it as experimental.

    btw. Do you know the ARRI-LUT-Generator? Try to build a pleasing one with this tool – Link

  13. Thank you chmee!

    I made some LUTs with the ARRI-LUT-Generator.
    I tried LOG to 709 which produces cube format and can be imported to RESOLVE.
    The result is dark so that i have to push the lum curve by half.
    The LUT with Knee and Shoulder at 1 is very hard and highlights are blown out. With Knee and Shoulder at 3 results are better.

    Whats irritating me is that 12 and 16bit conversion in raw2cdng is producing linear LUTs.
    While the Arri and other LUTs are Log to 709.

    I tried the option to produce a Linear to 709 conversion in Arris Generator.
    But the result is a ‚.data‘ file which i think can not be used in RESOLVE??

  14. @elise
    generally i do not recalculate the values in 12 and 16bit, i only add 2 or delete 2 bits – so this means, the data from the canon-sensor must be linear. so, i made a short try – rename the ending to .dat (instead of .data) and put it into the support/LUT/Arri-Path. then you can use it by rightclicking the footage and selecting the new LUT.

    For other readers interested in this LUT-Thing i recommend the Whitepaper from Arri. LINK

  15. Hello chmee,

    i renamed the LUT from the Arri Lut Generator to .dat
    The LUT get displayed in the Davinci Resolve Dialog.
    But it doesn´t affect the image. No changes?

  16. Hi,I found a bug .I shoot 25fps by 5dmk3.And i use V1.3.0 to make DNG .. But in dacini resolve ,I set resolve at 25fps. but in resolve MEDIA ,the file fps is 24.but V1.1.6 is OK~~ My english is bad~I am chinese. THANK YOU!

  17. Its not a bug. its a fixed value by me. In v1.1.6 i didnt set any framerate-values into the metadata. by now you can change the framerate inside resolve – read this thread.

    But: i will put this on todo-list for next version.

  18. This is by far the best converter for raw….but for some reason i cant import the full sequence in premiere, it only imports some frames, in resolve it opens up the entire sequence properly….im using ginger hdr in premiere….if that might help.

  19. Need help with corrupted raw file –
    Trying to recover a truncated raw file that has no footer. Is it possible to manually edit the footer information the is displayed in the batch window? I hope it will be possible to decode if I can manually input the correct frame size etc.?
    Thanks so much for your good work.

  20. sorry, its not my intention to help with corrupted files. try to append a selfwritten footer, then try again with my tool. (but, the idea of a footer-regenerator-tool is not that bad 🙂

  21. Cannot get it to work correctly. Works till the end, crashes… (window give error ’something got wrong‘). Log file doesn’t record the last stage.

    Is there a limit of 10Gb or so? Looks like it ’stalls‘ at 10Gb, since it’s makes DNG’s, but skips the last ones….

    Further, it’s nice that I can see the whole thing directly in Premiere, but how do I edit the ‚RAW‘ features there?? Perhaps it’s told somewhere, but after reading RAW stuff for two days, seeing numerous tutorials which only tell the story half, or not complete, I’m getting a bit ‚word blind‘ reading stuff about RAW video (ie. it’s a pain).

    let me know why the program crashes, and if it works, and with RAW editing somewhere, I donate stuff to you, because this would save me a shitload of time (comparison, the ‚old‘ way in After Effects, is still running, already rendering for 2 hours now, while the made DNG with your program, I already can see them in Premiere 😉 ..

    thnx mate, much appreciated.

  22. thanks for info. it’s time to make tests with files /strips bigger than 8gb..

    second point:
    i thought as well, adobe as inventor of the „universal“ raw-format dng would write such an importer. but no. its „specialized“ on BMCC-Files, so the only way is to emulate bmcc-structure – its not that hard, i just have to do it next days..

    but the third point:
    its pain. when imported, adobes says, its in an useable 10/12bit-workflow with full bitdepth support, but i didnt get it till now, while working with it (nor premiere neither speedgrade via directlink) it never felt like an greater bitdepth.

    by now i wanna fix bigfiles, goodlookin-import into premiere – but it seems, the only way to drive the fast lane is to work with proxies inside premiere and grading with an edl/xml in speedgrade. thats my rating.

    Discussions on that you’ll find in the ml-community and in the adobe-forum.

    regards chmee

  23. Thanks for your great tool. It´s really excellent!

    I´m working in Assimilate Scratch 7. My workflow is:

    1. Converting in party mode with Gamma 1.0
    2. Import to Premiere CC 7.1
    3. Exporting FinalCut-XML
    4. Conforming in Scratch

    I´m shooting on a 5D II with 25fps. The problem is, that Premiere and Scratch seem to read a framerate of 24fps from the dng-Files. Is it right that the framerate is stored in the files? If yes, would it be possible to adjust the framerate manually before the conversion?

    May be its interesting for someone: To get the image processed the right way I use the following debayer settings in Scratch (Matrix disabled) on 16 linear converted DNGs: http://postimg.org/image/4ydq556v7/

  24. @zillo
    i’m quite sure i’m embedding the according fps-rate. in speedgrade and in resolve as well the timeline sets the framerate, not the footage. maybe thats in scratch too. (but i will check that next days)

  25. hello,

    how can I change the name of the camera used, inside the app?

    I need to change from 50D to Canon EOS 50D.

    Can anyone help me with this question? Otherwise I cant apply the VISIONLOG to the CDNG files…

    thank you very much

  26. PARTY TIME! NO MORE PINK FRAMES!

    Finally we can color correct and send this to Adobe Media Encoder without the need to detour to After Effects.

  27. Chmee, thank you for the advice.

    The thing is, i am not able to do that inside the app. Tryed every mean possible (to my knowlodge) and didnt found a way!!

    Can you please be more specific about what I have to do? Really bad situation…

  28. After you convert a set of raw files, and try to convert the same set, I get an error and it closes the program. Is the program limited to converting 1 raw file once?

  29. Hi,

    I am having problems importing CinemaDNG + WAV from 5D mark III into Resolve. The WAV and the video is being imported separately. The clip is recognized as 24p (but it is 23.976p) and the timecode is different in the video than in audio. Even I can change the clip from 24p to 23.976p after importing I cannot get the clip working together with the WAV.

    Any ideas?

  30. Would you consider sharing the LUT or formula used to map 12-bit linear RAW to 10-bit log ?
    I could not find it in the DNG specs.

  31. Love your work and my team uses your program to improve time to work.

    I and the Magic Lantern community would like if you could import the vertical line banding feature!

    Keep it up for all of us! Much appreciated!

    Thank you.

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert