Recherche avancée
Médias (91)
-
Corona Radiata
26 septembre 2011, par
Mis à jour : Septembre 2011
Langue : English
Type : Audio
-
Lights in the Sky
26 septembre 2011, par
Mis à jour : Septembre 2011
Langue : English
Type : Audio
-
Head Down
26 septembre 2011, par
Mis à jour : Septembre 2011
Langue : English
Type : Audio
-
Echoplex
26 septembre 2011, par
Mis à jour : Septembre 2011
Langue : English
Type : Audio
-
Discipline
26 septembre 2011, par
Mis à jour : Septembre 2011
Langue : English
Type : Audio
-
Letting You
26 septembre 2011, par
Mis à jour : Septembre 2011
Langue : English
Type : Audio
Autres articles (74)
-
Les vidéos
21 avril 2011, parComme les documents de type "audio", Mediaspip affiche dans la mesure du possible les vidéos grâce à la balise html5 .
Un des inconvénients de cette balise est qu’elle n’est pas reconnue correctement par certains navigateurs (Internet Explorer pour ne pas le nommer) et que chaque navigateur ne gère en natif que certains formats de vidéos.
Son avantage principal quant à lui est de bénéficier de la prise en charge native de vidéos dans les navigateur et donc de se passer de l’utilisation de Flash et (...) -
Participer à sa traduction
10 avril 2011Vous pouvez nous aider à améliorer les locutions utilisées dans le logiciel ou à traduire celui-ci dans n’importe qu’elle nouvelle langue permettant sa diffusion à de nouvelles communautés linguistiques.
Pour ce faire, on utilise l’interface de traduction de SPIP où l’ensemble des modules de langue de MediaSPIP sont à disposition. ll vous suffit de vous inscrire sur la liste de discussion des traducteurs pour demander plus d’informations.
Actuellement MediaSPIP n’est disponible qu’en français et (...) -
Websites made with MediaSPIP
2 mai 2011, parThis page lists some websites based on MediaSPIP.
Sur d’autres sites (3949)
-
ffmpeg command to copy raw data into MP4 hangs and never finishes
28 mai 2019, par James AdamsI am running an
ffmpegcommand to copy raw H.265 data into an MP4 file and the command never completes. The file isn’t large, just 10 seconds worth of data. Here’s the command I’m running :$ ffmpeg -rtsp_transport tcp -i rtsp://user:pass@71.185.124.195:554/c1/b1558830329/e1558830339/replay/ -vcodec copy -y test_clip.mp4I then get output like this :
ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libavresample 3. 7. 0 / 3. 7. 0
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100
libpostproc 54. 7.100 / 54. 7.100
Guessed Channel Layout for Input Stream #0.1 : mono
Input #0, rtsp, from 'rtsp://user:pass@71.85.104.195:554/c1/b1558830329/e1558830339/replay/':
Metadata:
title : ONVIF RTSP Server
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 1920x1080, 30 fps, 30 tbr, 90k tbn, 60 tbc
Stream #0:1: Audio: pcm_mulaw, 8000 Hz, mono, s16, 64 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (pcm_mulaw (native) -> aac (native))
Press [q] to stop, [?] for help
[aac @ 0x55b71ce31900] Too many bits 8832.000000 > 6144 per frame requested, clamping to max
Output #0, mp4, to 'test_clip.mp4':
Metadata:
title : ONVIF RTSP Server
encoder : Lavf57.83.100
Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709, progressive), 1920x1080, q=2-31, 30 fps, 30 tbr, 90k tbn, 90k tbc
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, mono, fltp, 48 kb/s
Metadata:
encoder : Lavc57.107.100 aac
[mp4 @ 0x55b71ce17e00] Non-monotonous DTS in output stream 0:0; previous: 18000, current: 3000; changing to 18001. This may result in incorrect timestamps in the output file.
[mp4 @ 0x55b71ce17e00] Non-monotonous DTS in output stream 0:0; previous: 18001, current: 6000; changing to 18002. This may result in incorrect timestamps in the output file.
[mp4 @ 0x55b71ce17e00] Non-monotonous DTS in output stream 0:0; previous: 18002, current: 9000; changing to 18003. This may result in incorrect timestamps in the output file.
[mp4 @ 0x55b71ce17e00] Non-monotonous DTS in output stream 0:0; previous: 18003, current: 12000; changing to 18004. This may result in incorrect timestamps in the output file.
[mp4 @ 0x55b71ce17e00] Non-monotonous DTS in output stream 0:0; previous: 18004, current: 15000; changing to 18005. This may result in incorrect timestamps in the output file.
[mp4 @ 0x55b71ce17e00] Non-monotonous DTS in output stream 0:0; previous: 18005, current: 18000; changing to 18006. This may result in incorrect timestamps in the output file.
frame= 44 fps=0.0 q=-1.0 size= 256kB time=00:00:01.43 bitrate=1463.4kbits/frame= 60 fps= 57 q=-1.0 size= 512kB time=00:00:01.96 bitrate=2132.9kbits/frame= 76 fps= 48 q=-1.0 size= 768kB time=00:00:02.50 bitrate=2516.7kbits/frame= 92 fps= 44 q=-1.0 size= 1024kB time=00:00:03.03 bitrate=2765.6kbits/frame= 108 fps= 41 q=-1.0 size= 1024kB time=00:00:03.56 bitrate=2352.0kbits/[NULL @ 0x55b71cdfa540] SEI type 5 size 2208 truncated at 1944
frame= 123 fps= 39 q=-1.0 size= 1280kB time=00:00:04.06 bitrate=2578.6kbits/frame= 139 fps= 38 q=-1.0 size= 1536kB time=00:00:04.60 bitrate=2735.5kbits/frame= 155 fps= 37 q=-1.0 size= 1536kB time=00:00:05.13 bitrate=2451.3kbits/frame= 171 fps= 36 q=-1.0 size= 1792kB time=00:00:05.66 bitrate=2590.7kbits/frame= 187 fps= 36 q=-1.0 size= 2048kB time=00:00:06.20 bitrate=2706.1kbits/frame= 203 fps= 35 q=-1.0 size= 2304kB time=00:00:06.73 bitrate=2803.2kbits/frame= 219 fps= 35 q=-1.0 size= 2304kB time=00:00:07.26 bitrate=2597.4kbits/frame= 235 fps= 34 q=-1.0 size= 2560kB time=00:00:07.80 bitrate=2688.7kbits/frame= 246 fps= 33 q=-1.0 size= 2560kB time=00:00:08.16 bitrate=2568.0kbits/frame= 267 fps= 34 q=-1.0 size= 3072kB time=00:00:08.86 bitrate=2838.3kbits/frame= 282 fps= 34 q=-1.0 size= 3072kB time=00:00:09.36 bitrate=2686.8kbits/frame= 298 fps= 33 q=-1.0 size= 3328kB time=00:00:09.90 bitrate=2753.9kbits/frame= 314 fps= 33 q=-1.0 size= 3328kB time=00:00:10.43 bitrate=2613.1kbits/^Cspeed=1.11xThe command never completes and I need to kill it using
Ctrl-C.I have also tried adding the options
-nostdin -loglevel errorand appending this to the end of the command :> /dev/null 2>&1 < /dev/nullbut to no avail.I am testing the above since it mimics the actual code I’m developing which utilizes a Python package that wraps calls to
ffmpeg. The Python code below works well on a laptop but hangs on EC2 (both are Ubuntu 18.04) :import argparse
import datetime
import ffmpeg
# ------------------------------------------------------------------------------
if __name__ == "__main__":
# USAGE
# $ python collect_and_store.py --rtsp rtsp://user:pass1@71.85.125.110:554 \
# --duration 30 --count 10
# construct the argument parser and parse the arguments
args_parser = argparse.ArgumentParser()
args_parser.add_argument("--rtsp",
required=True,
type=str,
help="RTSP URL for video stream")
args_parser.add_argument("--duration",
required=True,
type=int,
help="duration of saved clips (in seconds)")
args_parser.add_argument("--count",
required=True,
type=int,
help="number of clips to save")
args = vars(args_parser.parse_args())
# sanity check for some of the arguments
if not args["rtsp"].lower().startswith("rtsp://"):
raise ValueError("Invalid input URL -- only RTSP supported")
seconds_per_clip = args["duration"]
start = int(datetime.datetime.now().strftime("%s"))
end = start + seconds_per_clip
number_of_files_to_collect = args["count"]
while number_of_files_to_collect > 0:
# build URL with start and end times
# NOTE URL is for Uniview RTSP, add options for other camera types
url = args["rtsp"] + f"/c1/b{start}/e{end}/replay/"
# file where we'll write clip data
temp_file = f"clip_b{start}_e{end}.mp4"
# create the equivalent of the ffmpeg command:
# $ ffmpeg -i -vcodec copy -y -rtsp_transport tcp
stream = ffmpeg.input(url)
stream = ffmpeg.output(stream, temp_file,
**{"codec:v": "copy",
"rtsp_transport": "tcp",
"y": None
}
)
ffmpeg.run(stream)
print(f"\n\nMP4 file created: {temp_file}")
number_of_files_to_collect -= 1
start = end + 1
end = start + seconds_per_clip -
How to Read DJI H264 FPV Feed as OpenCV Mat Object ?
29 mai 2019, par Walter MorawaTDLR : All DJI developers would benefit from decoding raw H264 video stream byte arrays to a format compatible with OpenCV.
I’ve spent a lot of time looking for a solution to reading DJI’s FPV feed as an OpenCV Mat object. I am probably overlooking something fundamental, since I am not too familiar with Image Encoding/Decoding.
Future developers who come across it will likely run into a bunch of the same issues I had. It would be great if DJI developers could use opencv directly without needing a 3rd party library.
I’m willing to use ffmpeg or JavaCV if necessary, but that’s quite the hurdle for most Android developers as we’re going to have to use cpp, ndk, terminal for testing, etc. That seems like overkill. Both options seem quite time consuming. This JavaCV H264 conversion seems unnecessarily complex. I found it from this relevant question.
I believe the issue lies in the fact that we need to decode both the byte array of length 6 (info array) and the byte array with current frame info simultaneously.
Basically, DJI’s FPV feed comes in a number of formats.
- Raw H264 (MPEG4) in VideoFeeder.VideoDataListener
// The callback for receiving the raw H264 video data for camera live view
mReceivedVideoDataListener = new VideoFeeder.VideoDataListener() {
@Override
public void onReceive(byte[] videoBuffer, int size) {
//Log.d("BytesReceived", Integer.toString(videoStreamFrameNumber));
if (videoStreamFrameNumber++%30 == 0){
//convert video buffer to opencv array
OpenCvAndModelAsync openCvAndModelAsync = new OpenCvAndModelAsync();
openCvAndModelAsync.execute(videoBuffer);
}
if (mCodecManager != null) {
mCodecManager.sendDataToDecoder(videoBuffer, size);
}
}
};- DJI also has it’s own Android decoder sample with FFMPEG to convert to YUV format.
@Override
public void onYuvDataReceived(final ByteBuffer yuvFrame, int dataSize, final int width, final int height) {
//In this demo, we test the YUV data by saving it into JPG files.
//DJILog.d(TAG, "onYuvDataReceived " + dataSize);
if (count++ % 30 == 0 && yuvFrame != null) {
final byte[] bytes = new byte[dataSize];
yuvFrame.get(bytes);
AsyncTask.execute(new Runnable() {
@Override
public void run() {
if (bytes.length >= width * height) {
Log.d("MatWidth", "Made it");
YuvImage yuvImage = saveYuvDataToJPEG(bytes, width, height);
Bitmap rgbYuvConvert = convertYuvImageToRgb(yuvImage, width, height);
Mat yuvMat = new Mat(height, width, CvType.CV_8UC1);
yuvMat.put(0, 0, bytes);
//OpenCv Stuff
}
}
});
}
}Edit : For those who want to see DJI’s YUV to JPEG function, here it is from the sample application :
private YuvImage saveYuvDataToJPEG(byte[] yuvFrame, int width, int height){
byte[] y = new byte[width * height];
byte[] u = new byte[width * height / 4];
byte[] v = new byte[width * height / 4];
byte[] nu = new byte[width * height / 4]; //
byte[] nv = new byte[width * height / 4];
System.arraycopy(yuvFrame, 0, y, 0, y.length);
Log.d("MatY", y.toString());
for (int i = 0; i < u.length; i++) {
v[i] = yuvFrame[y.length + 2 * i];
u[i] = yuvFrame[y.length + 2 * i + 1];
}
int uvWidth = width / 2;
int uvHeight = height / 2;
for (int j = 0; j < uvWidth / 2; j++) {
for (int i = 0; i < uvHeight / 2; i++) {
byte uSample1 = u[i * uvWidth + j];
byte uSample2 = u[i * uvWidth + j + uvWidth / 2];
byte vSample1 = v[(i + uvHeight / 2) * uvWidth + j];
byte vSample2 = v[(i + uvHeight / 2) * uvWidth + j + uvWidth / 2];
nu[2 * (i * uvWidth + j)] = uSample1;
nu[2 * (i * uvWidth + j) + 1] = uSample1;
nu[2 * (i * uvWidth + j) + uvWidth] = uSample2;
nu[2 * (i * uvWidth + j) + 1 + uvWidth] = uSample2;
nv[2 * (i * uvWidth + j)] = vSample1;
nv[2 * (i * uvWidth + j) + 1] = vSample1;
nv[2 * (i * uvWidth + j) + uvWidth] = vSample2;
nv[2 * (i * uvWidth + j) + 1 + uvWidth] = vSample2;
}
}
//nv21test
byte[] bytes = new byte[yuvFrame.length];
System.arraycopy(y, 0, bytes, 0, y.length);
for (int i = 0; i < u.length; i++) {
bytes[y.length + (i * 2)] = nv[i];
bytes[y.length + (i * 2) + 1] = nu[i];
}
Log.d(TAG,
"onYuvDataReceived: frame index: "
+ DJIVideoStreamDecoder.getInstance().frameIndex
+ ",array length: "
+ bytes.length);
YuvImage yuver = screenShot(bytes,Environment.getExternalStorageDirectory() + "/DJI_ScreenShot", width, height);
return yuver;
}
/**
* Save the buffered data into a JPG image file
*/
private YuvImage screenShot(byte[] buf, String shotDir, int width, int height) {
File dir = new File(shotDir);
if (!dir.exists() || !dir.isDirectory()) {
dir.mkdirs();
}
YuvImage yuvImage = new YuvImage(buf,
ImageFormat.NV21,
width,
height,
null);
OutputStream outputFile = null;
final String path = dir + "/ScreenShot_" + System.currentTimeMillis() + ".jpg";
try {
outputFile = new FileOutputStream(new File(path));
} catch (FileNotFoundException e) {
Log.e(TAG, "test screenShot: new bitmap output file error: " + e);
//return;
}
if (outputFile != null) {
yuvImage.compressToJpeg(new Rect(0,
0,
width,
height), 100, outputFile);
}
try {
outputFile.close();
} catch (IOException e) {
Log.e(TAG, "test screenShot: compress yuv image error: " + e);
e.printStackTrace();
}
runOnUiThread(new Runnable() {
@Override
public void run() {
displayPath(path);
}
});
return yuvImage;
}- DJI also appears to have a "getRgbaData" function, but there is literally not a single example online or by DJI. Go ahead and Google "DJI getRgbaData"... There’s only the reference to the api documentation that explains the self explanatory parameters and return values but nothing else. I couldn’t figure out where to call this and there doesn’t appear to be a callback function as there is with YUV. You can’t call it from the h264b byte array directly, but perhaps you can get it from the yuv data.
Option 1 is much more preferable to option 2, since YUV format has quality issues. Option 3 would also likely involve a decoder.
Here’s a screenshot that DJI’s own YUV conversion produces.
I’ve looked at a bunch of things about how to improve the YUV, remove green and yellow colors and whatnot, but at this point if DJI can’t do it right, I don’t want to invest resources there.
Regarding Option 1, I know there’s FFMPEG and JavaCV that seem like good options if I have to go the video decoding route.
Moreover, from what I understand, OpenCV can’t handle reading and writing video files without FFMPEG, but I’m not trying to read a video file, I am trying to read an H264/MPEG4 byte[] array. The following code seems to get positive results.
/* Async OpenCV Code */
private class OpenCvAndModelAsync extends AsyncTask {
@Override
protected double[] doInBackground(byte[]... params) {//Background Code Executing. Don't touch any UI components
//get fpv feed and convert bytes to mat array
Mat videoBufMat = new Mat(4, params[0].length, CvType.CV_8UC4);
videoBufMat.put(0,0, params[0]);
//if I add this in it says the bytes are empty.
//Mat videoBufMat = Imgcodecs.imdecode(encodeVideoBuf, Imgcodecs.IMREAD_ANYCOLOR);
//encodeVideoBuf.release();
Log.d("MatRgba", videoBufMat.toString());
for (int i = 0; i< videoBufMat.rows(); i++){
for (int j=0; j< videoBufMat.cols(); j++){
double[] rgb = videoBufMat.get(i, j);
Log.i("Matrix", "red: "+rgb[0]+" green: "+rgb[1]+" blue: "+rgb[2]+" alpha: "
+ rgb[3] + " Length: " + rgb.length + " Rows: "
+ videoBufMat.rows() + " Columns: " + videoBufMat.cols());
}
}
double[] center = openCVThingy(videoBufMat);
return center;
}
protected void onPostExecute(double[] center) {
//handle ui or another async task if necessary
}
}Rows = 4, Columns > 30k. I get lots of RGB values that seem valid, such as red = 113, green=75, blue=90, alpha=220 as a made up example ; however, I get a ton of 0,0,0,0 values. That should be somewhat okay, since Black is 0,0,0 (although I would have thought the alpha would be higher) and I have a black object in my image. I also don’t seem to get any white values 255, 255, 255, even though there is also plenty of white area. I’m not logging the entire byte so it could be there, but I have yet to see it.
However, when I try to compute the contours from this image, I almost always get that the moments (center x, y) are exactly in the center of the image. This error has nothing to do with my color filter or contours algorithm, as I wrote a script in python and tested that I implemented it correctly in Android by reading a still image and getting the exact same number of contours, position, etc in both Python and Android.
I noticed it has something to do with the videoBuffer byte size (bonus points if you can explain why every other length is 6)
2019-05-23 21:14:29.601 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 2425
2019-05-23 21:14:29.802 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 2659
2019-05-23 21:14:30.004 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:30.263 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6015
2019-05-23 21:14:30.507 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:30.766 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4682
2019-05-23 21:14:31.005 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:31.234 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 2840
2019-05-23 21:14:31.433 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4482
2019-05-23 21:14:31.664 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:31.927 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4768
2019-05-23 21:14:32.174 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:32.433 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4700
2019-05-23 21:14:32.668 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:32.864 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4740
2019-05-23 21:14:33.102 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:33.365 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4640My questions :
I. Is this the correct format to read an h264 byte as mat ?
Assuming the format is RGBA, that means row = 4 and columns = byte[].length, and CvType.CV_8UC4. Do I have height and width correct ? Something tells me YUV height and width is off. I was getting some meaningful results, but the contours were exactly in the center, just like with the H264.II. Does OpenCV handle MP4 in android like this ? If not, do we need to use FFMPEG or JavaCV ?
III. Does the int size have something to do with it ? Why is the int size occassionally 6, and other times 2400 to 6000 ? I’ve heard about the difference between this frames information and information about the next frame, but I’m simply not knowledgeable enough to know how to apply that here.
I’m starting to think this is where the issue lies. Since I need to get the 6 byte array for info about next frame, perhaps my modulo 30 is incorrect. So should I pass the 29th or 31st frame as a format byte for each frame ? How is that done in opencv or are we doomed to use the complicated ffmpeg ? How would I go about joining the neighboring frames/ byte arrays ?
IV. Can I fix this using Imcodecs ? I was hoping opencv would natively handle whether a frame was color from this frame or info about next frame. I added the below code, but I am getting an empty array :
Mat videoBufMat = Imgcodecs.imdecode(new MatOfByte(params[0]), Imgcodecs.IMREAD_UNCHANGED);This also is empty :
Mat encodeVideoBuf = new Mat(4, params[0].length, CvType.CV_8UC4);
encodeVideoBuf.put(0,0, params[0]);
Mat videoBufMat = Imgcodecs.imdecode(encodeVideoBuf, Imgcodecs.IMREAD_UNCHANGED);V. Should I try converting the bytes into Android jpeg and then import it ? Why is djis yuv decoder so complicated looking ? It makes me cautious from wanting to try ffmpeg or Javacv and just stick to Android decoder or opencv decoder.
VI. At what stage should I resize the frames to speed up calculations ?
Edit : DJI support got back to me and confirmed they don’t have any samples for doing what I’ve described. This is a time for we the community to make this available for everyone !
Upon further research, I don’t think opencv will be able to handle this as opencv’s android sdk has no functionality for video files/url’s (apart from a homegrown MJPEG codec).
So is there a way in Android to convert to mjpeg or similar in order to read ? In my application, I only need 1 or 2 frames per second, so perhaps I can save the image as jpeg.
But for real time applications we will likely need to write our own decoder. Please help so that we can make this available to everyone ! This question seems promising :
-
How to Read DJI FPV Feed as OpenCV Object ?
24 mai 2019, par Walter MorawaI’ve officially spent a lot of time looking for a solution to reading DJI’s FPV feed as an OpenCV Mat object. I am probably overlooking something simple, since I am not too familiar with Image Encoding/Decoding.
I apologize if I am missing something very basic, but I know I’m not the first person to have issues getting DJI’s FPV feed, and answering this question, especially if option 1 is possible, would be extremely valuable to many developers. Please consider upvoting this question, as I’ve thoroughly researched this issue and future developers who come across it will likely run into a bunch of the same issues I had.
I’m willing to use ffmpeg or Javacv if necessary, but that’s quite the hurdle for most Android developers as we’re going to have to use cpp, ndk, terminal for testing, etc. That seems like overkill.
I believe the issue lies in the fact that we need to decode both the byte array of length 6 (info array) and the byte array with current frame info simultaneously. Thanks in advance for your time.
Basically, DJI’s FPV feed comes in a number of formats.
- Raw H264 (MPEG4) in VideoFeeder.VideoDataListener
// The callback for receiving the raw H264 video data for camera live view
mReceivedVideoDataListener = new VideoFeeder.VideoDataListener() {
@Override
public void onReceive(byte[] videoBuffer, int size) {
//Log.d("BytesReceived", Integer.toString(videoStreamFrameNumber));
if (videoStreamFrameNumber++%30 == 0){
//convert video buffer to opencv array
OpenCvAndModelAsync openCvAndModelAsync = new OpenCvAndModelAsync();
openCvAndModelAsync.execute(videoBuffer);
}
if (mCodecManager != null) {
mCodecManager.sendDataToDecoder(videoBuffer, size);
}
}
};- DJI also has it’s own Android decoder sample with FFMPEG to convert to YUV format.
@Override
public void onYuvDataReceived(final ByteBuffer yuvFrame, int dataSize, final int width, final int height) {
//In this demo, we test the YUV data by saving it into JPG files.
//DJILog.d(TAG, "onYuvDataReceived " + dataSize);
if (count++ % 30 == 0 && yuvFrame != null) {
final byte[] bytes = new byte[dataSize];
yuvFrame.get(bytes);
AsyncTask.execute(new Runnable() {
@Override
public void run() {
if (bytes.length >= width * height) {
Log.d("MatWidth", "Made it");
YuvImage yuvImage = saveYuvDataToJPEG(bytes, width, height);
Bitmap rgbYuvConvert = convertYuvImageToRgb(yuvImage, width, height);
Mat yuvMat = new Mat(height, width, CvType.CV_8UC1);
yuvMat.put(0, 0, bytes);
//OpenCv Stuff
}
}
});
}
}- DJI also appears to have a "getRgbaData" function, but there is literally not a single example online or by DJI. Go ahead and Google "DJI getRgbaData"... There’s only the reference to the api documentation that explains the self explanatory parameters and return values but nothing else. I couldn’t figure out where to call this and there doesn’t appear to be a callback function as there is with YUV. You can’t call it from the h264b byte array directly, but perhaps you can get it from the yuv data.
Option 1 is much more preferable to option 2, since YUV format has quality issues. Option 3 would also likely involve a decoder.
Here’s a screenshot that DJI’s own YUV conversion produces.
I’ve looked at a bunch of things about how to improve the YUV, remove green and yellow colors and whatnot, but at this point if DJI can’t do it right, I don’t want to invest resources there.
Regarding Option 1, I know there’s FFMPEG and JavaCV that seem like good options if I have to go the video decoding route. However, both options seem quite time consuming. This JavaCV H264 conversion seems unnecessarily complex. I found it from this relevant question.
Moreover, from what I understand, OpenCV can’t handle reading and writing video files without FFMPEG, but I’m not trying to read a video file, I am trying to read an H264/MPEG4 byte[] array. The following code seems to get positive results.
/* Async OpenCV Code */
private class OpenCvAndModelAsync extends AsyncTask {
@Override
protected double[] doInBackground(byte[]... params) {//Background Code Executing. Don't touch any UI components
//get fpv feed and convert bytes to mat array
Mat videoBufMat = new Mat(4, params[0].length, CvType.CV_8UC4);
videoBufMat.put(0,0, params[0]);
//if I add this in it says the bytes are empty.
//Mat videoBufMat = Imgcodecs.imdecode(encodeVideoBuf, Imgcodecs.IMREAD_ANYCOLOR);
//encodeVideoBuf.release();
Log.d("MatRgba", videoBufMat.toString());
for (int i = 0; i< videoBufMat.rows(); i++){
for (int j=0; j< videoBufMat.cols(); j++){
double[] rgb = videoBufMat.get(i, j);
Log.i("Matrix", "red: "+rgb[0]+" green: "+rgb[1]+" blue: "+rgb[2]+" alpha: "
+ rgb[3] + " Length: " + rgb.length + " Rows: "
+ videoBufMat.rows() + " Columns: " + videoBufMat.cols());
}
}
double[] center = openCVThingy(videoBufMat);
return center;
}
protected void onPostExecute(double[] center) {
//handle ui or another async task if necessary
}
}Rows = 4, Columns > 30k. I get lots of RGB values that seem valid, such as red = 113, green=75, blue=90, alpha=220 as a made up example ; however, I get a ton of 0,0,0,0 values. That should be somewhat okay, since Black is 0,0,0 (although I would have thought the alpha would be higher) and I have a black object in my image.
However, when I try to compute the contours from this image, I almost always get that the moments (center x, y) are exactly in the center of the image. This error has nothing to do with my color filter or contours algorithm, as I wrote a script in python and tested that I implemented it correctly in Android by reading a still image and getting the exact same number of contours, position, etc in both Python and Android.
I noticed it has something to do with the videoBuffer byte size (bonus points if you can explain why every other length is 6 !)
2019-05-23 21:14:29.601 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 2425
2019-05-23 21:14:29.802 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 2659
2019-05-23 21:14:30.004 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:30.263 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6015
2019-05-23 21:14:30.507 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:30.766 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4682
2019-05-23 21:14:31.005 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:31.234 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 2840
2019-05-23 21:14:31.433 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4482
2019-05-23 21:14:31.664 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:31.927 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4768
2019-05-23 21:14:32.174 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:32.433 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4700
2019-05-23 21:14:32.668 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:32.864 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4740
2019-05-23 21:14:33.102 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 6
2019-05-23 21:14:33.365 21431-22086/com.dji.simulatorDemo D/VideoBufferSize: 4640My questions :
I. Is this the correct format way to read an h264 byte as mat ?
Assuming the format is RGBA, that means row = 4 and columns = byte[].length, and CvType.CV_8UC4. Do I have height and width correct ? Something tells me YUV height and width is off. I was getting some meaningful results, but the contours were exactly in the center, just like with the H264.II. Does OpenCV handle MP4 in android like this ? If not, do I need to use FFMPEG or JavaCV ?
III. Does the int size have something to do with it ? Why is the int size occassionally 6, and other times 2400 to 6000 ? I’ve heard about the difference between this frames information and information about the next frame, but I’m simply not knowledgeable enough to know how to apply that here.
I’m starting to think this is where the issue lies. Since I need to get the 6 byte array for info about next frame, perhaps my modulo 30 is incorrect. So should I pass the 29th or 31st frame as a format byte for each frame ? How is that done in opencv or are we doomed to use to the the complicated ffmpeg.IV. Can I fix this using Imcodecs ? I was hoping opencv would natively handle whether a frame was color from this frame or info about next frame. I added the below code, but I am getting an empty array :
Mat videoBufMat = Imgcodecs.imdecode(new MatOfByte(params[0]), Imgcodecs.IMREAD_UNCHANGED);This also is empty :
Mat encodeVideoBuf = new Mat(4, params[0].length, CvType.CV_8UC4);
encodeVideoBuf.put(0,0, params[0]);
Mat videoBufMat = Imgcodecs.imdecode(encodeVideoBuf, Imgcodecs.IMREAD_UNCHANGED);V. Should I try converting the bytes into Android jpeg and then import it ? Why is djis yuv decoder so complicated looking ? It makes me cautious from wanting to try ffmpeg or Javacv and just stick to Android decoder or opencv decoder.
VI. At what stage should I resize the frames to speed up calculations ?
Syndiquer tout le site
