Bug with encoding filenames to UTF-8 in jD3.2.16 [Fixed in 3.2.18]

[Arno]

Ähm...
what is with this?

I have found a simpler solution!
We should use Joomla standard transliteration method:

$lang = JLanguage::getInstance('ru-RU'); (in this case)
$filename = $lang->transliterate($filename);

and that is it!

[Makulia]

It is about integrating filename transliteration option.

ColinM suggest more detailed answer, so I have removed my post.
Idea is to use Joomla standard method to transliterate filenames from any lang.

Can you implement this code?

[Arno]

I will try it...

[Makulia]

Any progress about correcting filename saving into DB?
If you need any additional info, just let me know!

[Arno]

Sorry for the delay but i work extensive on the new jD content plugin.
But i had to wait until my test domains was moved from my hoster to another server with utf-8 support.  
This is now done so i can start now with some tests.

Edit:
Use you a windows 7 system for the file upload?
Frontend or backend creation form?
Which ftp program use you to check the upload result (filename) on the server?

[Arno]

Hi,
i have now the first results.

Server Test Environment:

jDownloads 3.2.16

PHP Built On                        Linux dd34424 3.2.0-67-generic #101-Ubuntu SMP Tue Jul 15 17:46:11 UTC 2014 x86_64
Database Version                5.5.40-nmm1-log
Database Collation                utf8_general_ci
PHP Version                        5.3.28-nmm1
Web Server                        Apache
WebServer to PHP Interface     fpm-fcgi
Joomla! Version                        Joomla! 3.3.6 Stable [ Ember ] 01-October-2014 02:00 GMT
Joomla! Platform Version        Joomla Platform 13.1.0 Stable [ Curiosity ] 24-Apr-2013 00:00 GMT
User Agent                        Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0

I have use this configuration settings:
- Create the directory name for a category automatically?  Yes
- Use UTF-8? Yes
- Replace spaces with underscores in names? No
- Change names to lowercase? No
- Remove/Change special characters in name?  No

For the upload i have renamed before a zip file to файл.zip with the Windows 7 File Explorer.
The new filename is correct viewed in the File Explorer.

Test Procedure:
1. go to backend 'downloads'
2. click on 'new'
3. type in a title and select as main file from the windows directory a file with name: файл.zip (see pic 1039a & 1039)
4. save the new download
5. result check in the new download self (see pic )
6. result check in the DB (see pic 1040)
7. result check on the server with the ftp program FileZilla 3.7.1 (see pic 1040b)
8. result check via WebFTP Script on the server (see pic 1040c)
9. go to the jDownloads in the frontend and try to download the file (see pic 1040d)
10. test the downloaded file with winzip to make sure that it is not corrupt (result okay)

I must say that i have only problems to view the filename with FileZilla. This seems not to support utf-8.
All other test results are for me okay!  

So i can not reproduce why you get wrong values in your DB. Maybe a wrong Database Collation?  

I will tomorrow test the 'folder creation process' with russian characters...

Edit:

@ Colin:

It processes a string and replaces all accented UTF-8 characters by unaccented ASCII-7 "equivalents", white spaces are replaced by hyphens and the string is lowercase.

Thanks for the hints. But I know this and i have already a modified version from this in uses: getCleanFolderFileName($str, $is_monitoring = false) in backend jdownloadshelper.php.

[gelöscht durch Administrator]

[Makulia]

Sorry for the delay but i work extensive on the new jD content plugin.
But i had to wait until my test domains was moved from my hoster to another server with utf-8 support.  
This is now done so i can start now with some tests.

Edit:
Use you a windows 7 system for the file upload?
Frontend or backend creation form?
Which ftp program use you to check the upload result (filename) on the server?

1) I have uploaded files from both Windows 8.1 and Ubuntu 14.10. Results were the same.
2) Fronted and backend. Problem is the same in both scenarios.
3) FTP|SFTP client - WinSCP

[Makulia]

Do yo have mbstring HTTP input encoding translation enable?

Could you please post your administrator/index.php?option=com_admin&view=sysinfo page
Especially interested in mbstring settings!
I think it may be something with php settings, that I am missing!

Thank you!

[Arno]

Relevant PHP Settings
Setting    Value

Safe Mode    Off
Open basedir    None
Display Errors    On
Short Open Tags    On
File Uploads    On
Magic Quotes    Off
Register Globals    Off
Output Buffering    Off
Session Save Path    /tmp
Session Auto Start    0
XML Enabled    Yes
Zlib Enabled    Yes
Native ZIP Enabled    Yes
Disabled Functions    None
Mbstring Enabled    Yes
Iconv Available    Yes

[Makulia]

I have done all actions the same way, as you did!

All this settings is the same as yours except disabled functions:
disable_functions   pcntl_alarm, pcntl_fork, pcntl_waitpid, pcntl_wait, pcntl_wifexited, pcntl_wifstopped, pcntl_wifsignaled, pcntl_wexitstatus, pcntl_wtermsig, pcntl_wstopsig, pcntl_signal, pcntl_signal_dispatch, pcntl_get_last_error, pcntl_strerror, pcntl_sigprocmask, pcntl_sigwaitinfo, pcntl_sigtimedwait, pcntl_exec, pcntl_getpriority, pcntl_setpriority

I have made some experiments with mb strings settings and it turns out there is some connections.
With this settings I was able to download and upload files (filename in the db is not trimmed), but file name is a mess. Please, look at the SH and send me this values of yours.

AddDefaultCharset UTF-8
php_value mbstring.language      Neutral   
php_value mbstring.internal_encoding   UTF-8   
php_value mbstring.encoding_translation   On      
php_value mbstring.http_input auto   
php_value mbstring.http_output      auto      
php_value mbstring.detect_order      auto      
#php_value mbstring.substitute_character   none      
php_value default_charset         UTF-8   


[gelöscht durch Administrator]

[Arno]

See pic from phpinfo

[gelöscht durch Administrator]

[Arno]

What has worked here exactly with Fabrik?
http://www.jdownloads.com/forum/index.php?topic=7528.msg29442#msg29442

[Makulia]

With your settings i am getting this:

Hmmm

[gelöscht durch Administrator]

[Makulia]

What has worked here exactly with Fabrik?
http://www.jdownloads.com/forum/index.php?topic=7528.msg29442#msg29442

Transliteration of names of uploaded files using Function stringURLSafe, provided by ColinM.

[Arno]

Transliteration of names of uploaded files using Function stringURLSafe, provided by ColinM.

What for a language have you activated in Joomla?