Answer 1

That looks like a file that was converted by native2ascii using the wrong parameters. To demonstrate, create a file with the contents

Gérer les modÚ

and save it as "a.txt" with the encoding UTF-8. Then run this command:

native2ascii -encoding windows-1252 a.txt b.txt

Open the new file and you should see this:

G\u00c3\u00a9rer les mod\u00c3\u0161

Now reverse the process, but specify ISO-8859-1 this time:

native2ascii -reverse -encoding ISO-8859-1 b.txt c.txt

Read the new file as UTF-8 and you should see this:

Gérer les modÀ\u0161

It recovers the "é" okay, but chokes on the "Ú", like your app did.

I don't know what all is going wrong in your app, but I'm pretty sure incorrect use of native2ascii is part of it. And that was probably the result of letting the app use the system default encoding. You should always specify the encoding when you save text, whether it's to a file or a database or what--never let it default. And if you don't have a good reason to choose something else, use UTF-8.

Answered by: Kimberly492 | Posted: 01-03-2022

Answer 2

Try to use

java -Dfile.encoding=UTF-8 ...

when starting the application in both systems.

Another way to solve the problem is to change the encoding from both system to UTF-8, but i prefer the first option (less intrusive on the system).

EDIT:

Check this answer on stackoverflow, It might help either:

Changing the default encoding for String(byte[])

Answered by: Adelaide284 | Posted: 01-03-2022

Answer 3

Instead of setting the system-wide character encoding, it might be easier and more robust, to specify the character encoding when reading and writing specific text data. How is your application reading the files? All the Java I/O package readers and writers support passing in a character encoding name to be used when reading/writing text to/from bytes. If you don't specify one, it will then use the platform default encoding, as you are likely experiencing.

Some databases are surprisingly limited in the text encodings they can accept. If your Java application reads the files as text, in the proper encoding, then it can output it to the database however it needs it. If your database doesn't support any encoding whose character repetoire includes the non-ASCII characters you have, then you may need to encode your non-English text first, for example into UTF-8 bytes, then Base64 encode those bytes as ASCII text.

PS: Never use String.getBytes() with no character encoding argument for exactly the reasons you are seeing.

Answered by: Blake215 | Posted: 01-03-2022

Answer 4

I managed to get past this error by running the command

export LC_ALL='en_GB.UTF-8'

This command set the locale for the shell that I was in. This set all of the LC_ environment variables to the Unicode file encoding.

Many thanks for all of your suggestions.

Answered by: Ned571 | Posted: 01-03-2022

Answer 5

You can also set the encoding at the command line, like so java -Dfile.encoding=utf-8.

Answered by: Roland387 | Posted: 01-03-2022

Answer 6

I think we'll need more information to be able to help you with your problem:

1. What exception are you getting exactly, and which method are you calling when it occurs.
2. What is the encoding of the input file? UTF8? UTF16/Unicode? ISO8859-1?

It'll also be helpful if you could provide us with relevant code snippets.

Also, a few things I want to point out:

1. The problem isn't occurring at the 'é' but later on.
2. It sounds like the character encoding may be hard coded in your application somewhere.

Answered by: Vivian225 | Posted: 01-03-2022

Answer 7

Also, you may want to verify that operating system packages to support UTF-8 (SUNWeulux, SUNWeuluf etc) are installed.

Answered by: Aida413 | Posted: 01-03-2022

Answer 8

Java uses operating system's default encoding while reading and writing files. Now, one should never rely on that. It's always a good practice to specify the encoding explicitly.

In Java you can use following for reading and writing:

Reading:

BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(inputPath),"UTF-8"));

Writing:

PrintWriter pw = new PrintWriter(new BufferedWriter(new OutputStreamWriter(new FileOutputStream(outputPath), "UTF-8")));

Answered by: Miranda908 | Posted: 01-03-2022

Similar questions

console application - How to output accented characters from Java on a Mac?

I'm trying to output accented characters from java into the Console app on a mac, however they just show up as ?'s. I've tried using System.console().writer() to output as well as just System.out, but they both produce the same result. It seems that maybe I need a different charset?

---

jsp - How to save Hindi Characters in the Application Properties file in Java?

We are trying to internationalize our Spring MVC web application in Hindi language. When we try to copy the Hindi text in the properties file, the properties file shows small boxes in places of Hindi characters. When we run the application and see the JSP, it shows questions marks (???????) in place of Hindi characters. Edit: My properties file has following contents.

---

html - Need to escape Special Characters in Java Web Application

I am working on a Java EE application with an Oracle DB. Now some content on the webpage has some special characters and I need to escape them. The characters are coming as show below: €˜T’ ! “One Chase.†$ % & ( ) ' / : ? ` — – _ ‚ " Test Is anyone aware what character encoding this is and how can I escape them? I need to escape them and repla...

---

How to make input text box to show hindi characters as typed on keyboard for a java application

It is a completed java desktop application and now need it to accept entries for data fields, input boxes, etc to be made in native language by non English speaking workers. So, essentially need a less invasive approach to actually internationalize the application to be usable in more than one localized regions. Asking my hurdles in step by step manner - what would be the ways to make a text input box show n...

---

java - How to display chinese characters in my rcp application?

I'm trying to internationalize my RCP app using the RCP i18n mechanism but i'm facing a problem : When I translate the String "Hello" in Arabic, I get "\u0645\u0631\u062D\u0628\u0627". I put this UTF-8 string in my .properties file and everything goes well. But when I translate the String "Hello" in Chinese, I get "\u4F60\u597D" and when I put this UTF-8 string in my .properties fi...

---

Arab/UTF8 Characters in Swing/Java application on HP-UX v11.3 (B11.31)

I am confronted with a strange situation that I do not understand. I run a Java-Swing test application, that reads Arab-UTF8 hard-coded strings, builds a simple JXTable and shows the UTF8 strings on a column. The application is an executable jar that is run with command java -cp test.jar org.test.MainTest If there is a need I can attach the code of the application. The application shows Arab ch...

---

special characters using barcode scanner to integrate in java application

I have a barcode Scanner and i want to integrate it on my JSF application, but unfortunetly when i scan a product I obtain a special chatacter in textfield like below : -&çé'à'"àà'éç -&çé'éçèàà&'" Could you please help me ?

---

java - Filtering an arraylist using first characters of string for Android Application

To study android development, I have created an android application called "Contacts", it's basically just a phonebook. I have a list of contacts with first name, last name and phone number. I also have an edittext which I use for filtering the list. Currently, I can filter the list by finding all entries containing the string inputted in the edittext. For example, I have 3 contacts: Jane Doe John D...

---

java - Allowing special characters which are not safe : Web Application

We recently had App Scan done on our application. As per fix for App Scan, we are not allowing \r`\n along with many other chars. Why are these chars dangerous? What if I have text box where users can type whatever they want? How to handle this scenarios?

---

encoding - How to display special characters in Java FX Application

I have an application in Java FX, I have the views in FXML files. The thing is that I have some labels with special characters as: " á,é,í,ó,ú,¿" and none of them are displayed well when I executed the application on a linux enviroment, but in Windows 8 all is fine. Following, I'll show some examples:

---

windows - IE6 generated strange worksheet name when doing export from java application

I am encountering error like test(10)[1].csv file cannot be found at C:\Documents and Settings\Ron\Local Settings\Temporary Internet Files\Content.IE5\PQ0STUVW When trying to do export of CSV file using the following codes. Anyone have any idea what could be wrong? This issue does not occur in IE7 / Firefox and is only specific to IE6. response.setContentType("applicati...

---

c# - Embedding Flash Player in a C++ or Java application?

I would like to embed Flash Player directly inside a C++ or Java application. I found an article that describes how to do this for C#: http://www.adobe.com/devnet/flash/articles/stock_history03.html Unfortunately, I have no experience with C#, COM or ActiveX. I need someone to translate this code to C++, allowing ...

---

java - How best to implement user selectable variables in web application

I have a Java based web-application and a new requirement to allow Users to place variables into text fields that are replaced when a document or other output is produced. How have others gone about this? I was thinking of having a pre-defined set of variables such as : @BOOKING_NUMBER@ @INVOICE_NUMBER@ Then when a user enters some text they can specify a variable inline ...

---

java - BIRT in a desktop application

Did someone ever used a BIRT report in a desktop application. I'm comming from the .NET environment and there you can use Crystal Reports to show reports in desktop apps. Is this possible with BIRT too, without having to set up a server environment? Can you give me some advice how to reach this goal? Thanks in advance.

---

which library better for faster java application swt or swing?

which library better for faster java application swt or swing?

---

java - Which is the best Open source application server?

---

java - Access spring bean that is exposed using http invoker from GWT application

Can I access spring bean that exposed using http invoker (server) from GWT application (client)? If so is there any example / tutorial for that?

---

How to close a Java Swing application from the code

What is the proper way to terminate a Swing application from the code, and what are the pitfalls? I'd tried to close my application automatically after a timer fires. But just calling dispose() on the JFrame didn't do the trick - the window vanished but the application did not terminate. However when closing the window with the close button, the application does terminate. What should I do...

---

java - How best can I isolate my application from an unreliable database?

I have a Java SOAP data service which sits on top of a Sybase database which, for reasons out of my control, has unreliable performance. The database is part of a vendor package which has been modified by an internal team and most of the issues are caused by slow response times at certain times of the day. The SOAP service provides data to a calculation grid and when I request data, I need the response time to be ...

---

java - Netbeans GUI Designer & Fixed-Size Application Panels

I'm having a problem, creating a fixed-size overall panel for a touchscreen GUI application that has to take up the entire screen. In a nutshell, the touchscreen is 800 x 600 pixels, and therefore I want the main GUI panel to be that size. When I start a new GUI project in NetBeans, I set the properties of the main panel for min/max/preferred size to 800 x 600, and the panel within the 'Design' view changes size. ...

---

Still can't find your answer? Check out these amazing Java communities for help...

Java Reddit Community | Java Help Reddit Community | Dev.to Java Community | Java Discord | Java Programmers (Facebook) | Java developers (Facebook)