Data Warehousing Community Forum
October 24, 2019, 01:17:40 am

Pages: [1]   Go Down
Author Topic: procedure in PL/SQL  (Read 3494 times)
DW Fresher

Reputation: +1/-0
Offline Offline

Posts: 9

View Profile
« on: December 13, 2007, 04:29:14 pm »


I have a CSV file.

The second field can occur more than once (I mean there are duplicate records)

So I need to extract only those duplicate records and place them into another file and delete them from original file.

In brief…I should count the no. of records for which the second field is equal. If the count is greater than 1 then I need to transfer all those lines into another file and delete from original file.

 Can anyone help me plz?
DW Fresher

Reputation: +2/-0
Offline Offline

Posts: 17

View Profile
« Reply #1 on: May 29, 2008, 12:24:15 am »

I understand you want to jot down the unique records and redirect them to another file.

There is command called uniq in unix which works on sorted data input. You could use the uniq command to eliminate or count duplicate lines. It works by comparing the previous and current line.

Let take your csv file to be duplicates.csv

Now consider the follwing:

cat duplicates|sort | uniq -c > unique.csv

The above would give the unique records along with the count of the number of times it occurred in the input.

using -d instead of -c would deletes duplicates and give only unique records.

If you want to work on a particular field/column only then you could use a combination of  cut and uniq commands.

Hope this helps you. I could elaborate more.  Smiley
Pages: [1]   Go Up
Jump to: